System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image of a meal taken by a user; means for analyzing the image to identify ingredients and dish information; means for storing the identified ingredients and dish information in a database; and means for managing the user's diet based on the stored ingredients and dish information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, health management and maintaining a balanced diet are important issues. However, many people find it difficult to accurately track their diet and maintain an appropriate nutritional balance in their busy daily lives. Furthermore, food ingredients and seasonings tend to run out, increasing the hassle of repurchasing them. Conventional methods require users to manually record their diet and manage their nutritional intake, which takes time and effort. This makes it difficult for users to maintain motivation to continuously manage their health. [Means for solving the problem]

[0005] The present invention provides a system that automatically analyzes images of meals taken by a user, identifies information about ingredients and dishes, and manages diets based on that information. This system also has a function that analyzes the user's meal records and consumed ingredients and suggests ingredients and seasonings that need to be repurchased. It also includes a function that automatically suggests nutritionally balanced recipes based on the user's meal records and preference data and notifies the user. This allows users to efficiently and sustainably manage their health and eliminates the need for repurchases. Specifically, the system includes the following means:

[0006] 1. A means for receiving images of food taken by a user.

[0007] 2. A means of analyzing the received image to identify ingredients and dish information.

[0008] 3. A means of storing the identified ingredients and dish information in a database.

[0009] 4. A means for managing the user's diet based on stored ingredient and dish information.

[0010] In addition, the system adds the following functions:

[0011] 1. A means of analyzing a user's food records and consumed ingredients to suggest ingredients and seasonings that need to be repurchased.

[0012] 2. Means of placing online orders based on the proposal.

[0013] 3. A means of suggesting recipes that take into account the user's nutritional balance based on the user's food records and preference data.

[0014] 4. A means of informing the user of suggested recipes.

[0015] This allows users to easily and efficiently manage their health in their daily lives and prevent shortages of ingredients and seasonings.

[0016] "User" refers to an individual who uses the system.

[0017] A "terminal" is a device operated by a user, and typically refers to a portable computer such as a smartphone or tablet.

[0018] "Server" refers to a remote computer system that receives and processes data sent from a user's terminal.

[0019] "Meal image" refers to a digital image generated when a user photographs their meal.

[0020] "Means for receiving images" refers to the function of the terminal to send images of food taken by the user to the server.

[0021] "Means for analyzing images" refers to algorithms or software that identify information about ingredients and dishes based on the images received by the server.

[0022] "Ingredient and dish information" refers to information such as ingredient names, dish names, quantities, and nutrients obtained through image analysis.

[0023] "Database" refers to a system for storing and managing information on identified ingredients and dishes.

[0024] "Means for dietary management" refers to a function that monitors and suggests a user's diet based on stored information on ingredients and dishes.

[0025] "Ingredients consumed" refers to the amount of ingredients used based on the user's food log.

[0026] "Ingredients and seasonings that need to be repurchased" refers to ingredients and seasonings that are running low and need to be purchased anew.

[0027] "Means of suggestion" refers to a function that notifies users of ingredients and seasonings that need to be repurchased, as well as recipes that take nutritional balance into consideration.

[0028] "Means for placing online orders" refers to a function that allows users to purchase ingredients and seasonings from an online shop based on suggestions.

[0029] "Preference data" refers to information about the user's favorite ingredients and dishes, as well as their past eating history.

[0030] "Means for suggesting recipes" refers to a function that generates new recipes that take nutritional balance into consideration based on the user's food records and preference data and notifies the user of them.

[0031] "Means of notification" refers to the function of displaying specific information or suggestions from the server on the terminal. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0033] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0034] First, the terms used in the following description will be explained.

[0035] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0036] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0037] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0038] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0039] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0040] [First embodiment]

[0041] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0042] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0043] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0044] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0045] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0046] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0047] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0048] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0049] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0051] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0052] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0053] The present invention relates to a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. An embodiment of this system will be described in detail below.

[0054] Meal photo recognition and management

[0055] User

[0056] The user takes a photo of the meal and launches the application, which saves the image of the meal on the user's device.

[0057] Terminal

[0058] The device sends the images taken by the user to the server, where the image data is compressed in an appropriate format to ensure efficient data transfer.

[0059] server

[0060] The server analyzes the images received from the device using machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify ingredients and dishes in the images.

[0061] The identified ingredients and dish information is stored in a database, which includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0062] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[0063] Specific examples

[0064] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[0065] Repurchase suggestions for ingredients and seasonings

[0066] server

[0067] The server periodically analyzes the user's food log and consumed food ingredient data, for example, to identify ingredients or condiments consumed in amounts above a certain threshold.

[0068] The server lists ingredients and seasonings that need to be repurchased and generates a suggestion message to notify the user at an appropriate time.

[0069] Terminal

[0070] The device then sends a repurchase suggestion message to the user, which includes the suggested ingredients and seasonings and a link to an online shop for repurchasing.

[0071] User

[0072] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[0073] Specific examples

[0074] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[0075] Recipe suggestions

[0076] server

[0077] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, including the user's preferred ingredients and past meal history.

[0078] The server creates a message to notify the user of the created recipe, which includes detailed information about the ingredients, quantities, cooking steps, etc.

[0079] Terminal

[0080] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[0081] User

[0082] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to follow a nutritionally balanced diet.

[0083] Specific examples

[0084] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[0085] conclusion

[0086] In this way, the system analyzes photos of meals taken by the user to manage their diet, suggests repurchases of consumed ingredients and seasonings, and provides nutritionally balanced recipes. This allows users to efficiently manage their health while reducing the effort required for managing and purchasing ingredients.

[0087] The processing flow will be explained below.

[0088] Meal photo recognition and management

[0089] Step 1:

[0090] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0091] Step 2:

[0092] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0093] Step 3:

[0094] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0095] Step 4:

[0096] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0097] Step 5:

[0098] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0099] Repurchase suggestions for ingredients and seasonings

[0100] Step 1:

[0101] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0102] Step 2:

[0103] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0104] Step 3:

[0105] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0106] Step 4:

[0107] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the ingredients and seasonings needed and a link to repurchase them.

[0108] Step 5:

[0109] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0110] Recipe suggestions

[0111] Step 1:

[0112] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0113] Step 2:

[0114] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0115] Step 3:

[0116] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0117] Step 4:

[0118] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0119] Step 5:

[0120] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0121] summary

[0122] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes, allowing them to efficiently manage their daily health.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] When users manage their daily diet, it is time-consuming to understand the details of ingredients and dishes and update records based on accurate nutritional information. Furthermore, knowing when to repurchase consumed ingredients and seasonings and suggesting recipes that take nutritional balance into consideration are also significant burdens for individual users. This makes ongoing health management difficult, creating a challenge.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes means for analyzing images of meals taken by the user to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for updating the user's meal record based on the stored information, means for analyzing the user's meal record and consumed ingredients and suggesting ingredients and seasonings that need to be repurchased, and means for suggesting recipes that take the user's nutritional balance into consideration based on the user's meal record and preference data. This allows the user to continuously and easily manage their diet, appropriately repurchase consumed ingredients and seasonings, and prepare meals that take nutritional balance into consideration.

[0128] A "user" is someone who uses the system to access functions such as dietary management, repurchasing ingredients, and recipe suggestions.

[0129] A "terminal" refers to an electronic device such as a smartphone, tablet, or PC used by a user to take photos of meals, send images, receive notifications, and so on.

[0130] A "server" is a computer system that analyzes image data sent from a terminal, stores the results in a database, and makes various suggestions to the user.

[0131] "Image data" refers to digital information of a photograph of a meal taken by a user, which is analyzed by the server.

[0132] "Machine learning algorithms" refer to artificial intelligence techniques used by the server to analyze image data, including convolutional neural networks (CNNs).

[0133] "Ingredients" are the individual food components in a meal that contain nutritional information.

[0134] A "dish" is a meal prepared according to a specific recipe and consisting of a combination of ingredients.

[0135] A "database" is an electronic recording device that stores information on ingredients and dishes, user meal records, preference data, and so on.

[0136] A "repurchase suggestion message" is a notification message sent by the server after analyzing the user's consumption data regarding ingredients or seasonings that need to be repurchased.

[0137] "Nutritional balance" refers to the distribution of each nutrient in the user's diet, and indicates whether it is balanced for health management.

[0138] "Recipe suggestions" refers to the server providing users with cooking methods that take nutritional balance into consideration based on the user's meal records and preference data.

[0139] "Notification" refers to the means by which a device communicates information such as repurchase suggestions or recipe suggestions to a user, and includes push notifications.

[0140] This invention relates to a system that analyzes images of meals taken by users, identifies ingredients and dishes, and performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. The system of the present invention is configured to achieve the following three main functions:

[0141] 1. Meal photo recognition and management

[0142] User

[0143] To record the contents of their meals, users take photos of their meals using the device's camera. The images are then saved directly to the device.

[0144] Terminal

[0145] The device has the ability to compress the image data captured by the user into an appropriate format (e.g., JPEG or PNG) and send it to the server. Specifically, it uses a compression algorithm to ensure efficient data transfer.

[0146] server

[0147] The server receives the image data sent from the device and analyzes it using a machine learning algorithm (e.g., convolutional neural network: CNN) to identify the ingredients and dishes.

[0148] The server stores the identified ingredients and dish information (e.g., ingredient name, dish name, portion size, calories, nutritional information) in a database.

[0149] Based on the saved information, it has the function of updating the user's food record and calculating and saving the total calories and nutritional components ingested.

[0150] Specific examples

[0151] A user takes a photo of their lunch: teriyaki chicken, rice, and salad. The device compresses the photo into JPEG format and sends it to the server. The server performs image analysis, identifies the ingredients (chicken, rice, lettuce, and tomato), and stores their nutritional information in a database. Finally, the user's food record is updated, enabling accurate nutritional management.

[0152] 2. Suggestions for repurchasing ingredients and seasonings

[0153] server

[0154] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[0155] The server generates a repurchase suggestion message regarding ingredients and seasonings that need to be repurchased and transmits it to the user's terminal.

[0156] Terminal

[0157] The device displays the repurchase suggestion message sent from the server to the user as a push notification, which includes the name of the suggested ingredient or seasoning and a link to the online shop for repurchase.

[0158] User

[0159] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[0160] Specific examples

[0161] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps a link to order soy sauce from an online shop.

[0162] 3. Recipe suggestions

[0163] server

[0164] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, and suggests appropriate recipes based on the user's preferred ingredients and past meal history.

[0165] Create a message to notify the user of the generated recipe, which includes detailed information such as ingredients, quantities, and cooking steps.

[0166] Terminal

[0167] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[0168] User

[0169] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to eat a nutritionally balanced diet.

[0170] Specific examples

[0171] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their iron.

[0172] conclusion

[0173] This system analyzes images of meals taken by users and provides dietary management, suggestions for repurchasing ingredients, and recipe suggestions based on nutritional balance, allowing users to efficiently manage their health and reducing the effort required for managing and purchasing ingredients, making ongoing health management easier.

[0174] Prompt Sentence Examples

[0175] "Please explain in natural language the program for a system that takes a photo of a meal using an app, sends it to a server, identifies ingredients and dishes from the analysis results, and updates the food record based on the nutritional information. Please include specific examples."

[0176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0177] Step 1:

[0178] The user takes a photo of their meal, and the image is saved in the device's internal storage.

[0179] Input: A user-taken image of a meal.

[0180] How it works: You take a photo using the camera app on your smartphone or tablet, which creates an image file.

[0181] Output: Image file saved in the device's internal storage.

[0182] Step 2:

[0183] The device compresses the image data and sends it to the server. The device uses a compression format such as JPEG or PNG.

[0184] Input: An image file stored in the device's internal storage.

[0185] How it works: An image compression algorithm is run on the device to generate a compressed image file, which is then sent over the internet to a server.

[0186] Output: Compressed image data sent to the server.

[0187] Step 3:

[0188] The server analyzes the received image data and uses machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify the ingredients and dishes in the image.

[0189] Input: Compressed image data sent from the terminal.

[0190] How it works: Machine learning algorithms run on the server to analyze the image, identify ingredients and dishes, and extract information about them.

[0191] Output: Information on identified ingredients and dishes (ingredients, dish name, portion size, calories, nutritional information, etc.).

[0192] Step 4:

[0193] The server stores the identified ingredients and dish information in a database.

[0194] Input: Ingredient and dish information identified by the analysis.

[0195] Operation: The analysis results are written and saved in the server's database system.

[0196] Output: Ingredient and dish information stored in a database.

[0197] Step 5:

[0198] The server updates the user's food record based on the stored information, calculates the total calories and nutritional information, and updates the database.

[0199] Input: Identified ingredients and dish information.

[0200] How it works: The server runs a nutrition calculation algorithm that adds calorie and nutrient totals to each user's food log.

[0201] Output: Updated user's meal log.

[0202] Step 6:

[0203] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[0204] Input: User's food log and consumed ingredients data.

[0205] How it works: The server runs a data analysis algorithm to identify ingredients and seasonings that are being consumed most frequently.

[0206] Output: A list of ingredients and seasonings that need to be repurchased.

[0207] Step 7:

[0208] The server generates a repurchase offer message and sends it to the user's terminal.

[0209] Input: A list of ingredients and seasonings that need to be repurchased.

[0210] Behavior: The server generates a proposal message and sends it to the device as a push notification.

[0211] Output: Repurchase offer message sent to the user's device.

[0212] Step 8:

[0213] The user checks the repurchase suggestion message and orders ingredients and seasonings from the online shop as needed.

[0214] Input: Repurchase suggestion message.

[0215] Action: The user sees the notification and taps the link to go to the online store and order the ingredients and seasonings they need.

[0216] Output: The ingredients and seasonings for the completed order.

[0217] Step 9:

[0218] The server analyzes the user's food records and preference data and generates recipes that take nutritional balance into consideration.

[0219] Input: User's food log and preference data.

[0220] How it works: Data analysis is performed on the server to generate nutritionally balanced recipes that are optimal for the user.

[0221] Output: Nutritionally balanced recipes.

[0222] Step 10:

[0223] The server notifies the user of the generated recipe.

[0224] Input: The generated recipe.

[0225] Operation: The server generates a recipe notification message and sends it to the device as a push notification.

[0226] Output: Recipe notification sent to the user's device.

[0227] Step 11:

[0228] The user checks the recipe notification and cooks using the suggested recipe as a reference.

[0229] Enter: recipe notification.

[0230] What happens: The user opens the notification, prepares the ingredients, and follows the recipe displayed to cook the food.

[0231] Output: A nutritionally balanced diet.

[0232] The above is the flow of processing of the program of this system.

[0233] (Application example 1)

[0234] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] In modern society, it is important for users to manage their health on a daily basis, but it is difficult to efficiently manage their diet and adjust nutritional balance in a busy lifestyle. There is also a lack of comprehensive systems that support overall dietary habits, such as managing food purchases and suggesting new recipes. Therefore, there is a need for a system that can identify information about ingredients and dishes based on photos of meals taken by users, and then use that information to manage their diet, suggest repurchasing ingredients, suggest recipes that take nutritional balance into consideration, and even deliver health-related content.

[0236] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0237] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified ingredient and dish information in a database, a means for managing the user's diet based on the stored ingredient and dish information, and a means for delivering health-related content. This allows users to automate their daily diet management and live a nutritionally balanced and healthy lifestyle. Furthermore, the server includes a means for suggesting ingredients and seasonings that need to be repurchased, a means for suggesting recipes based on the user's preference data, and a means for suggesting personalized health content using a generative AI model, enabling efficient comprehensive health management.

[0238] "User" refers to any individual or organization that uses this system.

[0239] "Meal image" refers to a digital image taken by a user that visually records the contents of a meal.

[0240] "Analysis" refers to the process of identifying ingredients and dish information from received food images, primarily using machine learning algorithms.

[0241] "Ingredient and dish information" is data indicating the components of a meal identified through image analysis, and includes the names of ingredients, dish names, quantities, nutritional components, etc.

[0242] "Database" refers to a digital storage system that organizes and stores information about identified ingredients and dishes so that it can be retrieved and updated later.

[0243] "Diet management" refers to the activity of tracking a user's diet and monitoring and adjusting the balance of calories and nutrients ingested.

[0244] "Health-related content" refers to digital content such as videos and articles that provide customized information and advice based on a user's health and diet.

[0245] "Repurchase suggestions" refers to notifications or messages that identify ingredients or seasonings that are missing based on the user's meal records and suggest purchasing them.

[0246] "Online ordering" refers to the process by which a user purchases suggested ingredients and seasonings through an online shopping system.

[0247] "Preference data" refers to personal data about ingredients or dishes selected by the user in the past, or the user's preferences.

[0248] "Recipe suggestion" refers to the activity of informing users of nutritionally balanced recipes based on their food records and preference data.

[0249] "Generative AI model" refers to the artificial intelligence algorithm used to analyze a user's eating habits and suggest personalized health content.

[0250] "Personalized health content" refers to health-related information and advice that is customized for a particular user based on the user's individual food records and nutritional status.

[0251] This invention provides a system for managing meals by analyzing images of meals taken by users and identifying information about ingredients and dishes. This system aims to provide comprehensive health management, including suggestions for repurchasing ingredients, recipe suggestions that take nutritional balance into consideration, and the distribution of health-related content.

[0252] 1. System Overview

[0253] The system consists of three main parts: user terminals, servers, and databases.

[0254] 2. User Device

[0255] The user device is a smartphone or tablet with a camera. The user takes a photo of the meal and launches a dedicated application. This application has the function of acquiring the captured image, compressing it in an appropriate format, and sending it to the server.

[0256] 3. Server

[0257] The server has the following functions:

[0258] Image analysis function

[0259] The server analyzes the received meal images using a convolutional neural network (CNN) to identify the ingredients and dishes in the image. The analysis results include the names of the ingredients, the dish, the serving size, calories, and other nutritional information.

[0260] Database Management Functions

[0261] The identified ingredients and dish information are stored in a database that holds the user's meal record, preference data, and data on ingredients consumed.

[0262] Meal management function

[0263] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[0264] Repurchase suggestion function

[0265] The server analyzes the user's meal log and identifies ingredients and seasonings that need to be repurchased. Once identified, the server notifies the user and provides a link to encourage online ordering.

[0266] Recipe suggestion function

[0267] The server analyzes the user's food records and preference data, and generates recipes that take into account the user's nutritional balance. The generated recipes are then notified to the user.

[0268] Health-related content distribution function

[0269] Using a generative AI model, it analyzes users' eating habits and suggests personalized health-related content, including videos and articles related to staying healthy.

[0270] 4. Specific Examples

[0271] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to a server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is stored in the user's database. The server updates the user's food record based on the nutritional information of these ingredients, providing appropriate nutritional management. The server also identifies ingredients that are in short supply and generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" The recipe suggestion function then sends the user a notification saying, "You are low on iron. We suggest a recipe using spinach and beef." The generative AI model also analyzes the user's eating habits and delivers "advice videos for increasing specific nutrients."

[0272] 5. Examples of Prompt Statements

[0273] "Based on photos of meals taken by users with their smartphones, please use a CNN model to analyze what ingredients and dishes are included, and upload the results to a database on the cloud. Also, please make sure that dietary management and recipe suggestions are performed based on the user's nutritional balance."

[0274] This system enables individual users to lead a nutritionally balanced and healthy life while automating their daily dietary management, and also enables efficient comprehensive health management.

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] The user takes a photo of the meal.

[0278] Input: Food image

[0279] Output: Raw image files saved on the device

[0280] Specific operation: The user takes a photo of the meal using the camera app on their smartphone and launches a dedicated application.

[0281] Step 2:

[0282] The device sends the captured image to the server.

[0283] Input: Image file saved on the device

[0284] Output: Compressed image data sent to the server

[0285] Specific operation: The application compresses the image file in the appropriate format and sends it to the server using a data transfer protocol (e.g., HTTP).

[0286] Step 3:

[0287] The server analyzes the received image.

[0288] Input: Compressed image data sent

[0289] Output: Analysis results (information on identified ingredients and dishes)

[0290] How it works: The server uses a convolutional neural network (CNN) to analyze the image and identify ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0291] Step 4:

[0292] The server stores the identified ingredients and dish information in a database.

[0293] Input: Analysis results (identified ingredients and dish information)

[0294] Output: Ingredient and dish information stored in a database

[0295] Specific operation: The server executes queries to store the analysis results in an SQL or NoSQL database.

[0296] Step 5:

[0297] The server manages dietary management based on the stored information.

[0298] Input: Ingredient and dish information stored in a database

[0299] Output: Data required for dietary management, updated user diet record

[0300] Specific operation: The server retrieves the stored information, calculates the total calories and nutritional intake, and updates the user's food record.

[0301] Step 6:

[0302] The server analyzes the user's meal log and the ingredients consumed.

[0303] Input: User's food records and consumed ingredients data stored in a database

[0304] Output: List of ingredients and seasonings that need to be repurchased

[0305] Specific operation: The server uses an algorithm to analyze the user's food records and identify ingredients and seasonings whose consumption exceeds a certain threshold.

[0306] Step 7:

[0307] The server generates a repurchase offer message and sends it to the terminal.

[0308] Input: List of ingredients and seasonings that need to be repurchased

[0309] Output: Repurchase suggestion message

[0310] Specific operation: The server generates a proposal message and sends it to the user's device via push notification or email.

[0311] Step 8:

[0312] The server analyzes the user's food records and preference data.

[0313] Input: Ingredient and dish information stored in the database, user preference data

[0314] Output: Generate recipes that take nutritional balance into consideration

[0315] Specific operation: The server generates nutritionally balanced recipes based on the user's preference data and food records, and uses a generative AI model to suggest personalized health content.

[0316] Step 9:

[0317] The device will notify you of suggested recipes and health-related content.

[0318] Input: Generated recipes and personalized health content

[0319] Output: Notifications and application screen display

[0320] Specific behavior: The device displays the received notification and allows the user to view recipes and health-related content.

[0321] These processing steps enable users to efficiently manage their diet and lead a nutritionally balanced and healthy life. Furthermore, the generative AI model provides personalized health content, allowing users to receive advice based on their individual health conditions.

[0322] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0323] The present invention is a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes that take nutritional balance into consideration based on this information. By further combining this with an emotion engine, the system makes suggestions based on the user's emotions. An embodiment of this system will be described in detail below.

[0324] Meal photo recognition and management

[0325] User

[0326] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0327] Terminal

[0328] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0329] server

[0330] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0331] The identified ingredients and dish information is stored in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0332] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0333] Specific examples

[0334] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[0335] Repurchase suggestions for ingredients and seasonings

[0336] server

[0337] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0338] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0339] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0340] Terminal

[0341] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[0342] User

[0343] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0344] Specific examples

[0345] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[0346] Recipe suggestions

[0347] server

[0348] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0349] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0350] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0351] Terminal

[0352] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0353] User

[0354] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0355] Specific examples

[0356] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[0357] Incorporating an emotion engine

[0358] server

[0359] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[0360] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[0361] Terminal

[0362] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[0363] User

[0364] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[0365] Specific examples

[0366] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[0367] conclusion

[0368] This invention provides a unique function that combines an emotion engine with a system that analyzes images of meals taken by users to manage their diet, suggest repurchasing ingredients, and suggest recipes based on nutritional balance. This enables health management according to the user's emotional state and makes more personalized suggestions.

[0369] The processing flow will be explained below.

[0370] Meal photo recognition and management

[0371] Step 1:

[0372] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0373] Step 2:

[0374] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0375] Step 3:

[0376] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0377] Step 4:

[0378] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0379] Step 5:

[0380] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0381] Repurchase suggestions for ingredients and seasonings

[0382] Step 1:

[0383] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0384] Step 2:

[0385] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0386] Step 3:

[0387] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0388] Step 4:

[0389] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the suggested ingredients and seasonings and a link to repurchase them.

[0390] Step 5:

[0391] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0392] Recipe suggestions

[0393] Step 1:

[0394] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0395] Step 2:

[0396] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0397] Step 3:

[0398] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0399] Step 4:

[0400] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0401] Step 5:

[0402] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0403] Incorporating an emotion engine

[0404] Step 1:

[0405] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[0406] Step 2:

[0407] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[0408] Step 3:

[0409] The server creates emotion-based suggestion messages, which include suggestions for ingredients and dishes based on the user's emotional state.

[0410] Step 4:

[0411] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[0412] Step 5:

[0413] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[0414] Specific examples

[0415] Example 1: Food photo recognition

[0416] A user takes a photo of "toast and coffee" for breakfast. The device sends the photo to the server, which analyzes the photo and identifies the toast and coffee. This nutritional information is stored in a database and the user's food record is updated.

[0417] Example 2: Repurchase offer

[0418] The server analyzes the user's food log and detects that milk consumption is increasing. The server generates a repurchase suggestion message saying "Milk stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order milk from the online shop.

[0419] Example 3: Recipe suggestions

[0420] The server analyzes the user's food records and detects a vitamin D deficiency. The server generates a nutritionally balanced recipe using fish and spinach and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their vitamin D.

[0421] Example 4: Emotion engine suggestions

[0422] While a user is using a smartphone app, the emotion engine recognizes that the user is tired. The server generates a suggestion message saying, "How about some relaxing herbal tea?" and sends it to the device. The user can check the notification and prepare the suggested herbal tea to relax.

[0423] conclusion

[0424] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes. Furthermore, by incorporating an emotion engine, suggestions can be made based on the user's emotional state, enabling even more personalized support.

[0425] Example 2

[0426] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0427] Conventional dietary management systems analyze nutritional information and suggest repurchasing ingredients, but are unable to provide suggestions that take the user's emotional state into account. As a result, personalized suggestions based on the user's mental state and preferences are insufficient, limiting the effectiveness of health management. The present invention aims to realize more personalized dietary and health management by recognizing the user's emotions in real time and adjusting the suggestions based on this.

[0428] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0429] In this invention, the server includes means for receiving images of meals taken by a user, means for analyzing the images to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for managing the user's meals based on the stored information about ingredients and dishes, means for recognizing the user's emotions from facial expressions, tone of voice, and text data, and means for adjusting the content of suggestions based on the recognized emotions, thereby enabling personalized suggestions based on the user's emotional state.

[0430] A "user" is an individual who uses the system to take photos of their meals and use those photos for analysis.

[0431] The "server" is a central processing unit that receives image data sent by users and performs analysis and proposals.

[0432] A "terminal" is a device used by a user, such as a smartphone or tablet, that has the function of sending captured images to a server.

[0433] A "meal image" is a photograph of a meal taken by a user, and serves as data for identifying ingredients and dishes.

[0434] "Analysis" refers to the process performed to identify ingredients and dish information from the received food image.

[0435] "Ingredient and dish information" refers to the names of ingredients, dishes, amounts, calories, and other nutritional information identified through analysis.

[0436] "Database" refers to an information management system for storing information on identified ingredients and dishes.

[0437] "Dietary management" refers to the act of managing the calorie and nutrient intake of a user based on stored information about ingredients and dishes.

[0438] "Means for recognizing emotions" refers to technology that has the ability to analyze a user's emotions from their facial expressions, tone of voice, and text data.

[0439] "Means to adjust suggestion content" refers to a function that appropriately changes the content of meal management, repurchase, and recipe suggestions based on recognized emotions.

[0440] The invention begins with a user taking a picture of a meal using a device such as a smartphone or tablet and sending the image to a server. The device compresses the image using a library such as Python's Pillow and sends it to the server in an appropriate format (e.g., JPEG or PNG). The server then inputs the received image data into an image recognition module, which uses a convolutional neural network (CNN) and an AI model.

[0441] The image recognition module uses machine learning algorithms to identify ingredient and dish information from the submitted image. The identified information includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information. This information is stored in a database on the server. This eliminates the need for users to record their meals themselves, allowing for automated diet management.

[0442] The present invention also incorporates an emotion engine, allowing the server to recognize emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state, and if the user is feeling stressed or tired, for example, it will suggest ingredients and dishes that have a relaxing effect.

[0443] For example, a user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. When the device sends the photo to the server, the server's image recognition module analyzes it and identifies the ingredients, such as chicken, rice, lettuce, and tomato. The nutritional information for these ingredients is then stored in a database, and the user's meal record is updated. Furthermore, the emotion engine analyzes the user's emotional state and, if necessary, suggests relaxing herbal teas.

[0444] Examples of prompt sentences include the following:

[0445] Ingredient identification model

[0446] "Identify the ingredients and name of the dish based on a photo of the meal taken by the user."

[0447] Emotion Engine

[0448] "Analyze the user's emotions from facial expressions, tone of voice, and text data, and generate appropriate suggestions."

[0449] In this way, by appropriately utilizing the generative AI model and prompt sentences, personalized suggestions based on the user's condition become possible. This system enables efficient and effective dietary management for users, and also realizes health management that takes into account the user's emotional state.

[0450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0451] Step 1: User takes and submits a photo of their meal

[0452] Specific actions and inputs

[0453] The user opens the smartphone app and takes a photo of their meal. This photo becomes the input data. Once the photo is taken, the user taps the "Send Photo" button.

[0454] Data processing and output

[0455] The device compresses the captured image and converts it to the appropriate format (e.g., JPEG or PNG). The image is compressed using the Python Pillow library and sent to the server. The compressed image data is the output.

[0456] Step 2: The server receives the image data and analyzes it.

[0457] Specific actions and inputs

[0458] The server receives the compressed image data sent from the terminal. The received image data is input data.

[0459] Data processing and output

[0460] The server's image recognition module analyzes the received image data and uses a convolutional neural network (CNN) to identify information about ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0461] Step 3: The server saves the analysis results to the database and updates the food record.

[0462] Specific actions and inputs

[0463] The analysis results, which are information on ingredients and dishes, become input data.

[0464] Data processing and output

[0465] The server stores the identified ingredients and dish information in a database. Based on this stored data, the user's food record is updated, including the total calories, protein, carbohydrates, and fat intake. The updated food record is the output.

[0466] Step 4: The server analyzes the meal record and suggests ingredients and seasonings that need to be repurchased.

[0467] Specific actions and inputs

[0468] The server receives the latest food record data as input.

[0469] Data processing and output

[0470] The server analyzes consumption over a specific period (e.g., one week) and identifies ingredients and seasonings that exceed a consumption threshold. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. This suggestion message is the output.

[0471] Step 5: The device sends a repurchase suggestion message to the user

[0472] Specific actions and inputs

[0473] The repurchase offer message sent from the server is the input data.

[0474] Data processing and output

[0475] The device sends a repurchase suggestion message to the user as a push notification. The user confirms the notification and is presented with a link to repurchase based on the suggestion. The notified suggestion message is the output.

[0476] Step 6: The server analyzes the food record and preference data and suggests recipes

[0477] Specific actions and inputs

[0478] The user's food records and preference data are the input data.

[0479] Data processing and output

[0480] The server analyzes this data and automatically generates a recipe that takes nutritional balance into consideration. It creates a recipe suggestion message that includes the necessary ingredients, quantities, and cooking steps. The generated recipe is the output.

[0481] Step 7: The device notifies the user of the recipe suggestion message

[0482] Specific actions and inputs

[0483] The recipe suggestion message sent from the server is the input data.

[0484] Data processing and output

[0485] The device sends a recipe suggestion message to the user as a push notification. The user can check the notification and view the recipe details. The notified recipe suggestion message is the output.

[0486] Step 8: The server recognizes the user's emotions using the emotion engine and adjusts the suggestions.

[0487] Specific actions and inputs

[0488] The input data includes the user's facial expression, tone of voice, and text data.

[0489] Data processing and output

[0490] The server uses an emotion engine to analyze the user's emotions from these data in real time. Based on the recognized emotions, the server adjusts the recommendations, such as ingredients and dishes that have a relaxing effect. The adjusted recommendation message is output.

[0491] Step 9: The device notifies the user with a sentiment-based suggestion message

[0492] Specific actions and inputs

[0493] The input data is a suggestion message based on the emotion sent from the server.

[0494] Data processing and output

[0495] The device sends a sentiment-based suggestion message to the user as a push notification. The user acknowledges the notification and is provided with guidance to take appropriate action. The notified suggestion message is the output.

[0496] Through the above steps, the system of the present invention realizes the user's dietary management, repurchase suggestions, recipe suggestions based on nutritional balance, and personalized health management according to the user's emotional state.

[0497] (Application example 2)

[0498] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0499] Conventional dietary management systems are limited to managing users' dietary records and nutritional balance, and have the problem of not being able to provide dietary suggestions based on the user's emotional state. This makes it difficult to manage health care that takes the user's emotional state into account, and there are issues with not being able to provide personalized suggestions. Furthermore, there is also the problem that repurchase suggestions and online ordering are not automated, making it difficult for users to efficiently replenish their food supplies.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0501] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified information on ingredients and dishes in a database, a means for managing the user's diet based on the stored information on ingredients and dishes, a means for recognizing emotions from the user's facial expressions, tone of voice, and text data, and a means for adjusting suggestions based on the recognized emotions. This allows for more refined diet management for the user and makes suggestions tailored to their emotions. Furthermore, it enables automated repurchase suggestions and online ordering, improving user convenience.

[0502] "Means for receiving images of meals taken by the user" is a function for collecting image data of meals taken using a device such as a smartphone or digital camera and sending it to a server or cloud.

[0503] The "means for analyzing the image and identifying information about ingredients and dishes" is a function that uses machine learning algorithms and image recognition technology to identify the types of ingredients and dishes contained in the image based on the received image data.

[0504] The "means for saving information on the identified ingredients and dishes in a database" is a function for recording and accumulating the analyzed ingredients and dish types and related nutritional information in a database.

[0505] "Means for managing the user's diet based on stored information on ingredients and dishes" refers to a function that uses information stored in the database to manage the user's dietary history, nutritional intake, calories, etc., and to suggest appropriate meals and manage health.

[0506] "Means for recognizing emotions from a user's facial expression, tone of voice, and text data" is a function for analyzing emotions from a user's facial expression data, tone of voice, and text content, and identifying that state.

[0507] The "means for adjusting the content of suggestions based on the recognized emotion" is a function for using the recognized emotion information to appropriately change the content of suggestions for meals and ingredients to the user, and providing services that correspond to the user's emotional state.

[0508] The "means for suggesting ingredients and seasonings that need to be repurchased" is a function that identifies ingredients and seasonings that are in short supply based on the user's meal record and information on ingredients consumed, and suggests to the user that they be repurchased.

[0509] The "means for placing an online order based on the suggestions" is a function that provides a link or interface for the user to place an order for the suggested ingredients or seasonings at an online shop.

[0510] The "means for providing links related to delivery services" is a function that provides users with links for ordering the suggested dishes and ingredients through delivery services.

[0511] In one embodiment of the present invention, a system is used that receives images of food taken by a user, identifies ingredients and dishes, and makes suggestions based on the user's emotional state. This system is realized using the user's smartphone, a server, a database, an emotion recognition engine, etc.

[0512] 1. Meal photo recognition and management

[0513] User

[0514] Users use a smartphone app to take a photo of their meal and tap the "send" button.

[0515] Terminal

[0516] The device compresses the captured image and sends it to the server, where the image data is sent in the appropriate format (e.g., JPEG or PNG).

[0517] server

[0518] The server inputs the received image data into an image recognition module and uses machine learning algorithms (e.g., CNN) to identify the ingredients and dishes in the image. The identified ingredients and dishes are stored in a database. This information includes the names of the ingredients, the names of the dishes, the serving sizes, calories, and other nutritional information. The stored information is also used to update the user's food log and manage calorie and macronutrient intake.

[0519] Specific examples

[0520] For example, a user takes a photo of their lunch of "teriyaki chicken," "rice," and "salad," and the device sends the photo to the server. The server analyzes the image, identifies the chicken, rice, lettuce, and tomato, and stores their nutritional information in the user's database. This data is used to manage the user's nutritional intake.

[0521] 2. Suggestions for repurchasing ingredients and seasonings

[0522] server

[0523] The server periodically analyzes the user's food log and checks the food ingredient data consumed. This analysis includes the amount consumed over a specific period (e.g., one week). If an ingredient or seasoning exceeds a threshold, it determines that it needs to be repurchased. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. The message includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0524] Terminal

[0525] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[0526] User

[0527] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0528] Specific examples

[0529] The server analyzes the recent food log and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[0530] 3. Recipe suggestions

[0531] server

[0532] The server analyzes the user's food records and preference data, including the user's favorite ingredients and nutritional balance data. It automatically generates recipes that take nutritional balance into consideration and selects recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences. It then creates a message to notify the user of the generated recipe. The message contains detailed information such as the recipe name, required ingredients, quantities, and cooking instructions.

[0533] Terminal

[0534] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0535] User

[0536] The user confirms the notification and views the details of the suggested recipe. The user then prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0537] Specific examples

[0538] The server analyzes the user's recent food log and detects that they are iron deficient. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user checks the notification and cooks the dish according to the recipe.

[0539] 4. Incorporating an Emotional Engine

[0540] server

[0541] The server uses an emotion engine that recognizes the user's emotions from their facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state and adjust the recommendations based on the recognized emotion. For example, if the user is feeling stressed, the server will suggest ingredients and dishes that have a relaxing effect.

[0542] Terminal

[0543] The terminal receives the emotion-based suggestion message from the server and notifies the user.

[0544] User

[0545] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[0546] Specific examples

[0547] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[0548] Prompt Sentence Examples

[0549] "Analyze the user's food photos and identify the food and ingredients. Also, determine the user's emotions from their facial expressions and tone of voice, and suggest ingredients and recipes based on that."

[0550] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0551] Step 1:

[0552] A user takes a photo of their meal using their smartphone and taps the "send" button.

[0553] Input: Image file of the food you have taken

[0554] Output: Food image data sent to the server

[0555] Specific operation: Take a photo of your meal using the smartphone's camera app, and send the image to the server using the "Send" button in the application.

[0556] Step 2:

[0557] The device compresses the images it takes and sends them to the server.

[0558] Input: Image file of the photographed meal

[0559] Output: Compressed food image data (JPEG or PNG format)

[0560] Specific operation: The device compresses the image file into the appropriate format and sends it to the specified server address.

[0561] Step 3:

[0562] The image data received by the server is input into an image recognition module to identify the ingredients and the dish.

[0563] Input: Compressed food image data

[0564] Output: Identified ingredients and dish information (e.g., ingredient name, dish name)

[0565] Specific operation: The received image data is input into a machine learning algorithm (e.g., a CNN model) to identify the ingredients and dishes in the image.

[0566] Step 4:

[0567] The server stores the identified ingredients and dish information in a database.

[0568] Input: Identified ingredients and dish information

[0569] Output: Updated food record in the database

[0570] Specific behavior: Writes the identified ingredients and dish data to a database and updates the user's meal record.

[0571] Step 5:

[0572] The server recognizes the user's emotions from their facial expressions, tone of voice, and text data.

[0573] Input: User's facial expression data, voice data, text data

[0574] Output: Perceived emotional state (e.g., stress, happiness)

[0575] Specific operation: Facial expressions and tone of voice collected by the smartphone's camera and microphone are input into an emotion recognition engine to identify the user's emotional state.

[0576] Step 6:

[0577] The server adjusts the suggestions based on the perceived sentiment.

[0578] Input: Perceived emotional state

[0579] Output: Tailored recommendations (e.g., ingredients and dishes with relaxing effects)

[0580] What it does: It applies algorithms that change the food and recipe suggestions provided to users based on their emotional state.

[0581] Step 7:

[0582] The server notifies the user of suggested recipes and repurchase suggestions.

[0583] Input: Adjusted recommendations, ingredients and seasonings that need to be repurchased

[0584] Output: A notification message displayed on the user's terminal.

[0585] Specific behavior: A suggestion message is sent to the user's smartphone as a push notification, and the user receives the notification.

[0586] Step 8:

[0587] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions.

[0588] Input: Notification content (recipe information, repurchase link)

[0589] Output: Order completed at online store

[0590] Specific behavior: The user taps the notification to be taken to the linked online shop, where they can add the suggested ingredients and seasonings to their cart and place an order.

[0591] Step 9:

[0592] The server generates a message suggesting ingredients and seasonings that need to be repurchased and sends it to the user.

[0593] Input: Food log and food consumption data

[0594] Output: Suggestion message (list of missing ingredients and seasonings)

[0595] Specific operation: Analyzes food records, identifies ingredients and seasonings consumed frequently, and generates suggestion messages based on that information and sends them to the user.

[0596] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0597] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0598] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0599] [Second embodiment]

[0600] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0601] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0602] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0603] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0604] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0605] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0606] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0607] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0608] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0609] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0610] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0611] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0612] The present invention relates to a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. An embodiment of this system will be described in detail below.

[0613] Meal photo recognition and management

[0614] User

[0615] The user takes a photo of the meal and launches the application, which saves the image of the meal on the user's device.

[0616] Terminal

[0617] The device sends the images taken by the user to the server, where the image data is compressed in an appropriate format to ensure efficient data transfer.

[0618] server

[0619] The server analyzes the images received from the device using machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify ingredients and dishes in the images.

[0620] The identified ingredients and dish information is stored in a database, which includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0621] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[0622] Specific examples

[0623] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[0624] Repurchase suggestions for ingredients and seasonings

[0625] server

[0626] The server periodically analyzes the user's food log and consumed food ingredient data, for example, to identify ingredients or condiments consumed in amounts above a certain threshold.

[0627] The server lists ingredients and seasonings that need to be repurchased and generates a suggestion message to notify the user at an appropriate time.

[0628] Terminal

[0629] The device then sends a repurchase suggestion message to the user, which includes the suggested ingredients and seasonings and a link to an online shop for repurchasing.

[0630] User

[0631] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[0632] Specific examples

[0633] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[0634] Recipe suggestions

[0635] server

[0636] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, including the user's preferred ingredients and past meal history.

[0637] The server creates a message to notify the user of the created recipe, which includes detailed information about the ingredients, quantities, cooking steps, etc.

[0638] Terminal

[0639] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[0640] User

[0641] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to follow a nutritionally balanced diet.

[0642] Specific examples

[0643] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[0644] conclusion

[0645] In this way, the system analyzes photos of meals taken by the user to manage their diet, suggests repurchases of consumed ingredients and seasonings, and provides nutritionally balanced recipes. This allows users to efficiently manage their health while reducing the effort required for managing and purchasing ingredients.

[0646] The processing flow will be explained below.

[0647] Meal photo recognition and management

[0648] Step 1:

[0649] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0650] Step 2:

[0651] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0652] Step 3:

[0653] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0654] Step 4:

[0655] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0656] Step 5:

[0657] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0658] Repurchase suggestions for ingredients and seasonings

[0659] Step 1:

[0660] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0661] Step 2:

[0662] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0663] Step 3:

[0664] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0665] Step 4:

[0666] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the ingredients and seasonings needed and a link to repurchase them.

[0667] Step 5:

[0668] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0669] Recipe suggestions

[0670] Step 1:

[0671] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0672] Step 2:

[0673] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0674] Step 3:

[0675] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0676] Step 4:

[0677] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0678] Step 5:

[0679] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0680] summary

[0681] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes, allowing them to efficiently manage their daily health.

[0682] Example 1

[0683] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] When users manage their daily diet, it is time-consuming to understand the details of ingredients and dishes and update records based on accurate nutritional information. Furthermore, knowing when to repurchase consumed ingredients and seasonings and suggesting recipes that take nutritional balance into consideration are also significant burdens for individual users. This makes ongoing health management difficult, creating a challenge.

[0685] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0686] In this invention, the server includes means for analyzing images of meals taken by the user to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for updating the user's meal record based on the stored information, means for analyzing the user's meal record and consumed ingredients and suggesting ingredients and seasonings that need to be repurchased, and means for suggesting recipes that take the user's nutritional balance into consideration based on the user's meal record and preference data. This allows the user to continuously and easily manage their diet, appropriately repurchase consumed ingredients and seasonings, and prepare meals that take nutritional balance into consideration.

[0687] A "user" is someone who uses the system to access functions such as dietary management, repurchasing ingredients, and recipe suggestions.

[0688] A "terminal" refers to an electronic device such as a smartphone, tablet, or PC used by a user to take photos of meals, send images, receive notifications, and so on.

[0689] A "server" is a computer system that analyzes image data sent from a terminal, stores the results in a database, and makes various suggestions to the user.

[0690] "Image data" refers to digital information of a photograph of a meal taken by a user, which is analyzed by the server.

[0691] "Machine learning algorithms" refer to artificial intelligence techniques used by the server to analyze image data, including convolutional neural networks (CNNs).

[0692] "Ingredients" are the individual food components in a meal that contain nutritional information.

[0693] A "dish" is a meal prepared according to a specific recipe and consisting of a combination of ingredients.

[0694] A "database" is an electronic recording device that stores information on ingredients and dishes, user meal records, preference data, and so on.

[0695] A "repurchase suggestion message" is a notification message sent by the server after analyzing the user's consumption data regarding ingredients or seasonings that need to be repurchased.

[0696] "Nutritional balance" refers to the distribution of each nutrient in the user's diet, and indicates whether it is balanced for health management.

[0697] "Recipe suggestions" refers to the server providing users with cooking methods that take nutritional balance into consideration based on the user's meal records and preference data.

[0698] "Notification" refers to the means by which a device communicates information such as repurchase suggestions or recipe suggestions to a user, and includes push notifications.

[0699] This invention relates to a system that analyzes images of meals taken by users, identifies ingredients and dishes, and performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. The system of the present invention is configured to achieve the following three main functions:

[0700] 1. Meal photo recognition and management

[0701] User

[0702] To record the contents of their meals, users take photos of their meals using the device's camera. The images are then saved directly to the device.

[0703] Terminal

[0704] The device has the ability to compress the image data captured by the user into an appropriate format (e.g., JPEG or PNG) and send it to the server. Specifically, it uses a compression algorithm to ensure efficient data transfer.

[0705] server

[0706] The server receives the image data sent from the device and analyzes it using a machine learning algorithm (e.g., convolutional neural network: CNN) to identify the ingredients and dishes.

[0707] The server stores the identified ingredients and dish information (e.g., ingredient name, dish name, portion size, calories, nutritional information) in a database.

[0708] Based on the saved information, it has the function of updating the user's food record and calculating and saving the total calories and nutritional components ingested.

[0709] Specific examples

[0710] A user takes a photo of their lunch: teriyaki chicken, rice, and salad. The device compresses the photo into JPEG format and sends it to the server. The server performs image analysis, identifies the ingredients (chicken, rice, lettuce, and tomato), and stores their nutritional information in a database. Finally, the user's food record is updated, enabling accurate nutritional management.

[0711] 2. Suggestions for repurchasing ingredients and seasonings

[0712] server

[0713] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[0714] The server generates a repurchase suggestion message regarding ingredients and seasonings that need to be repurchased and transmits it to the user's terminal.

[0715] Terminal

[0716] The device displays the repurchase suggestion message sent from the server to the user as a push notification, which includes the name of the suggested ingredient or seasoning and a link to the online shop for repurchase.

[0717] User

[0718] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[0719] Specific examples

[0720] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps a link to order soy sauce from an online shop.

[0721] 3. Recipe suggestions

[0722] server

[0723] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, and suggests appropriate recipes based on the user's preferred ingredients and past meal history.

[0724] Create a message to notify the user of the generated recipe, which includes detailed information such as ingredients, quantities, and cooking steps.

[0725] Terminal

[0726] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[0727] User

[0728] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to eat a nutritionally balanced diet.

[0729] Specific examples

[0730] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their iron.

[0731] conclusion

[0732] This system analyzes images of meals taken by users and provides dietary management, suggestions for repurchasing ingredients, and recipe suggestions based on nutritional balance, allowing users to efficiently manage their health and reducing the effort required for managing and purchasing ingredients, making ongoing health management easier.

[0733] Prompt Sentence Examples

[0734] "Please explain in natural language the program for a system that takes a photo of a meal using an app, sends it to a server, identifies ingredients and dishes from the analysis results, and updates the food record based on the nutritional information. Please include specific examples."

[0735] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0736] Step 1:

[0737] The user takes a photo of their meal, and the image is saved in the device's internal storage.

[0738] Input: A user-taken image of a meal.

[0739] How it works: You take a photo using the camera app on your smartphone or tablet, which creates an image file.

[0740] Output: Image file saved in the device's internal storage.

[0741] Step 2:

[0742] The device compresses the image data and sends it to the server. The device uses a compression format such as JPEG or PNG.

[0743] Input: An image file stored in the device's internal storage.

[0744] How it works: An image compression algorithm is run on the device to generate a compressed image file, which is then sent over the internet to a server.

[0745] Output: Compressed image data sent to the server.

[0746] Step 3:

[0747] The server analyzes the received image data and uses machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify the ingredients and dishes in the image.

[0748] Input: Compressed image data sent from the terminal.

[0749] How it works: Machine learning algorithms run on the server to analyze the image, identify ingredients and dishes, and extract information about them.

[0750] Output: Information on identified ingredients and dishes (ingredients, dish name, portion size, calories, nutritional information, etc.).

[0751] Step 4:

[0752] The server stores the identified ingredients and dish information in a database.

[0753] Input: Ingredient and dish information identified by the analysis.

[0754] Operation: The analysis results are written and saved in the server's database system.

[0755] Output: Ingredient and dish information stored in a database.

[0756] Step 5:

[0757] The server updates the user's food record based on the stored information, calculates the total calories and nutritional information, and updates the database.

[0758] Input: Identified ingredients and dish information.

[0759] How it works: The server runs a nutrition calculation algorithm that adds calorie and nutrient totals to each user's food log.

[0760] Output: Updated user's meal log.

[0761] Step 6:

[0762] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[0763] Input: User's food log and consumed ingredients data.

[0764] How it works: The server runs a data analysis algorithm to identify ingredients and seasonings that are being consumed most frequently.

[0765] Output: A list of ingredients and seasonings that need to be repurchased.

[0766] Step 7:

[0767] The server generates a repurchase offer message and sends it to the user's terminal.

[0768] Input: A list of ingredients and seasonings that need to be repurchased.

[0769] Behavior: The server generates a proposal message and sends it to the device as a push notification.

[0770] Output: Repurchase offer message sent to the user's device.

[0771] Step 8:

[0772] The user checks the repurchase suggestion message and orders ingredients and seasonings from the online shop as needed.

[0773] Input: Repurchase suggestion message.

[0774] Action: The user sees the notification and taps the link to go to the online store and order the ingredients and seasonings they need.

[0775] Output: The ingredients and seasonings for the completed order.

[0776] Step 9:

[0777] The server analyzes the user's food records and preference data and generates recipes that take nutritional balance into consideration.

[0778] Input: User's food log and preference data.

[0779] How it works: Data analysis is performed on the server to generate nutritionally balanced recipes that are optimal for the user.

[0780] Output: Nutritionally balanced recipes.

[0781] Step 10:

[0782] The server notifies the user of the generated recipe.

[0783] Input: The generated recipe.

[0784] Operation: The server generates a recipe notification message and sends it to the device as a push notification.

[0785] Output: Recipe notification sent to the user's device.

[0786] Step 11:

[0787] The user checks the recipe notification and cooks using the suggested recipe as a reference.

[0788] Enter: recipe notification.

[0789] What happens: The user opens the notification, prepares the ingredients, and follows the recipe displayed to cook the food.

[0790] Output: A nutritionally balanced diet.

[0791] The above is the flow of processing of the program of this system.

[0792] (Application example 1)

[0793] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0794] In modern society, it is important for users to manage their health on a daily basis, but it is difficult to efficiently manage their diet and adjust nutritional balance in a busy lifestyle. There is also a lack of comprehensive systems that support overall dietary habits, such as managing food purchases and suggesting new recipes. Therefore, there is a need for a system that can identify information about ingredients and dishes based on photos of meals taken by users, and then use that information to manage their diet, suggest repurchasing ingredients, suggest recipes that take nutritional balance into consideration, and even deliver health-related content.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0796] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified ingredient and dish information in a database, a means for managing the user's diet based on the stored ingredient and dish information, and a means for delivering health-related content. This allows users to automate their daily diet management and live a nutritionally balanced and healthy lifestyle. Furthermore, the server includes a means for suggesting ingredients and seasonings that need to be repurchased, a means for suggesting recipes based on the user's preference data, and a means for suggesting personalized health content using a generative AI model, enabling efficient comprehensive health management.

[0797] "User" refers to any individual or organization that uses this system.

[0798] "Meal image" refers to a digital image taken by a user that visually records the contents of a meal.

[0799] "Analysis" refers to the process of identifying ingredients and dish information from received food images, primarily using machine learning algorithms.

[0800] "Ingredient and dish information" is data indicating the components of a meal identified through image analysis, and includes the names of ingredients, dish names, quantities, nutritional components, etc.

[0801] "Database" refers to a digital storage system that organizes and stores information about identified ingredients and dishes so that it can be retrieved and updated later.

[0802] "Diet management" refers to the activity of tracking a user's diet and monitoring and adjusting the balance of calories and nutrients ingested.

[0803] "Health-related content" refers to digital content such as videos and articles that provide customized information and advice based on a user's health and diet.

[0804] "Repurchase suggestions" refers to notifications or messages that identify ingredients or seasonings that are missing based on the user's meal records and suggest purchasing them.

[0805] "Online ordering" refers to the process by which a user purchases suggested ingredients and seasonings through an online shopping system.

[0806] "Preference data" refers to personal data about ingredients or dishes selected by the user in the past, or the user's preferences.

[0807] "Recipe suggestion" refers to the activity of informing users of nutritionally balanced recipes based on their food records and preference data.

[0808] "Generative AI model" refers to the artificial intelligence algorithm used to analyze a user's eating habits and suggest personalized health content.

[0809] "Personalized health content" refers to health-related information and advice that is customized for a particular user based on the user's individual food records and nutritional status.

[0810] This invention provides a system for managing meals by analyzing images of meals taken by users and identifying information about ingredients and dishes. This system aims to provide comprehensive health management, including suggestions for repurchasing ingredients, recipe suggestions that take nutritional balance into consideration, and the distribution of health-related content.

[0811] 1. System Overview

[0812] The system consists of three main parts: user terminals, servers, and databases.

[0813] 2. User Device

[0814] The user device is a smartphone or tablet with a camera. The user takes a photo of the meal and launches a dedicated application. This application has the function of acquiring the captured image, compressing it in an appropriate format, and sending it to the server.

[0815] 3. Server

[0816] The server has the following functions:

[0817] Image analysis function

[0818] The server analyzes the received meal images using a convolutional neural network (CNN) to identify the ingredients and dishes in the image. The analysis results include the names of the ingredients, the dish, the serving size, calories, and other nutritional information.

[0819] Database Management Functions

[0820] The identified ingredients and dish information are stored in a database that holds the user's meal record, preference data, and data on ingredients consumed.

[0821] Meal management function

[0822] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[0823] Repurchase suggestion function

[0824] The server analyzes the user's meal log and identifies ingredients and seasonings that need to be repurchased. Once identified, the server notifies the user and provides a link to encourage online ordering.

[0825] Recipe suggestion function

[0826] The server analyzes the user's food records and preference data, and generates recipes that take into account the user's nutritional balance. The generated recipes are then notified to the user.

[0827] Health-related content distribution function

[0828] Using a generative AI model, it analyzes users' eating habits and suggests personalized health-related content, including videos and articles related to staying healthy.

[0829] 4. Specific Examples

[0830] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to a server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is stored in the user's database. The server updates the user's food record based on the nutritional information of these ingredients, providing appropriate nutritional management. The server also identifies ingredients that are in short supply and generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" The recipe suggestion function then sends the user a notification saying, "You are low on iron. We suggest a recipe using spinach and beef." The generative AI model also analyzes the user's eating habits and delivers "advice videos for increasing specific nutrients."

[0831] 5. Examples of Prompt Statements

[0832] "Based on photos of meals taken by users with their smartphones, please use a CNN model to analyze what ingredients and dishes are included, and upload the results to a database on the cloud. Also, please make sure that dietary management and recipe suggestions are performed based on the user's nutritional balance."

[0833] This system enables individual users to lead a nutritionally balanced and healthy life while automating their daily dietary management, and also enables efficient comprehensive health management.

[0834] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0835] Step 1:

[0836] The user takes a photo of the meal.

[0837] Input: Food image

[0838] Output: Raw image files saved on the device

[0839] Specific operation: The user takes a photo of the meal using the camera app on their smartphone and launches a dedicated application.

[0840] Step 2:

[0841] The device sends the captured image to the server.

[0842] Input: Image file saved on the device

[0843] Output: Compressed image data sent to the server

[0844] Specific operation: The application compresses the image file in the appropriate format and sends it to the server using a data transfer protocol (e.g., HTTP).

[0845] Step 3:

[0846] The server analyzes the received image.

[0847] Input: Compressed image data sent

[0848] Output: Analysis results (information on identified ingredients and dishes)

[0849] How it works: The server uses a convolutional neural network (CNN) to analyze the image and identify ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0850] Step 4:

[0851] The server stores the identified ingredients and dish information in a database.

[0852] Input: Analysis results (identified ingredients and dish information)

[0853] Output: Ingredient and dish information stored in a database

[0854] Specific operation: The server executes queries to store the analysis results in an SQL or NoSQL database.

[0855] Step 5:

[0856] The server manages dietary management based on the stored information.

[0857] Input: Ingredient and dish information stored in a database

[0858] Output: Data required for dietary management, updated user diet record

[0859] Specific operation: The server retrieves the stored information, calculates the total calories and nutritional intake, and updates the user's food record.

[0860] Step 6:

[0861] The server analyzes the user's meal log and the ingredients consumed.

[0862] Input: User's food records and consumed ingredients data stored in a database

[0863] Output: List of ingredients and seasonings that need to be repurchased

[0864] Specific operation: The server uses an algorithm to analyze the user's food records and identify ingredients and seasonings whose consumption exceeds a certain threshold.

[0865] Step 7:

[0866] The server generates a repurchase offer message and sends it to the terminal.

[0867] Input: List of ingredients and seasonings that need to be repurchased

[0868] Output: Repurchase suggestion message

[0869] Specific operation: The server generates a proposal message and sends it to the user's device via push notification or email.

[0870] Step 8:

[0871] The server analyzes the user's food records and preference data.

[0872] Input: Ingredient and dish information stored in the database, user preference data

[0873] Output: Generate recipes that take nutritional balance into consideration

[0874] Specific operation: The server generates nutritionally balanced recipes based on the user's preference data and food records, and uses a generative AI model to suggest personalized health content.

[0875] Step 9:

[0876] The device will notify you of suggested recipes and health-related content.

[0877] Input: Generated recipes and personalized health content

[0878] Output: Notifications and application screen display

[0879] Specific behavior: The device displays the received notification and allows the user to view recipes and health-related content.

[0880] These processing steps enable users to efficiently manage their diet and lead a nutritionally balanced and healthy life. Furthermore, the generative AI model provides personalized health content, allowing users to receive advice based on their individual health conditions.

[0881] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0882] The present invention is a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes that take nutritional balance into consideration based on this information. By further combining this with an emotion engine, the system makes suggestions based on the user's emotions. An embodiment of this system will be described in detail below.

[0883] Meal photo recognition and management

[0884] User

[0885] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0886] Terminal

[0887] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0888] server

[0889] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0890] The identified ingredients and dish information is stored in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0891] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0892] Specific examples

[0893] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[0894] Repurchase suggestions for ingredients and seasonings

[0895] server

[0896] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0897] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0898] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0899] Terminal

[0900] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[0901] User

[0902] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0903] Specific examples

[0904] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[0905] Recipe suggestions

[0906] server

[0907] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0908] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0909] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0910] Terminal

[0911] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0912] User

[0913] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0914] Specific examples

[0915] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[0916] Incorporating an emotion engine

[0917] server

[0918] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[0919] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[0920] Terminal

[0921] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[0922] User

[0923] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[0924] Specific examples

[0925] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[0926] conclusion

[0927] This invention provides a unique function that combines an emotion engine with a system that analyzes images of meals taken by users to manage their diet, suggest repurchasing ingredients, and suggest recipes based on nutritional balance. This enables health management according to the user's emotional state and makes more personalized suggestions.

[0928] The processing flow will be explained below.

[0929] Meal photo recognition and management

[0930] Step 1:

[0931] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[0932] Step 2:

[0933] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[0934] Step 3:

[0935] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[0936] Step 4:

[0937] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[0938] Step 5:

[0939] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[0940] Repurchase suggestions for ingredients and seasonings

[0941] Step 1:

[0942] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[0943] Step 2:

[0944] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[0945] Step 3:

[0946] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[0947] Step 4:

[0948] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the suggested ingredients and seasonings and a link to repurchase them.

[0949] Step 5:

[0950] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[0951] Recipe suggestions

[0952] Step 1:

[0953] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[0954] Step 2:

[0955] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[0956] Step 3:

[0957] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[0958] Step 4:

[0959] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[0960] Step 5:

[0961] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[0962] Incorporating an emotion engine

[0963] Step 1:

[0964] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[0965] Step 2:

[0966] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[0967] Step 3:

[0968] The server creates emotion-based suggestion messages, which include suggestions for ingredients and dishes based on the user's emotional state.

[0969] Step 4:

[0970] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[0971] Step 5:

[0972] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[0973] Specific examples

[0974] Example 1: Food photo recognition

[0975] A user takes a photo of "toast and coffee" for breakfast. The device sends the photo to the server, which analyzes the photo and identifies the toast and coffee. This nutritional information is stored in a database and the user's food record is updated.

[0976] Example 2: Repurchase offer

[0977] The server analyzes the user's food log and detects that milk consumption is increasing. The server generates a repurchase suggestion message saying "Milk stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order milk from the online shop.

[0978] Example 3: Recipe suggestions

[0979] The server analyzes the user's food records and detects a vitamin D deficiency. The server generates a nutritionally balanced recipe using fish and spinach and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their vitamin D.

[0980] Example 4: Emotion engine suggestions

[0981] While a user is using a smartphone app, the emotion engine recognizes that the user is tired. The server generates a suggestion message saying, "How about some relaxing herbal tea?" and sends it to the device. The user can check the notification and prepare the suggested herbal tea to relax.

[0982] conclusion

[0983] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes. Furthermore, by incorporating an emotion engine, suggestions can be made based on the user's emotional state, enabling even more personalized support.

[0984] Example 2

[0985] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0986] Conventional dietary management systems analyze nutritional information and suggest repurchasing ingredients, but are unable to provide suggestions that take the user's emotional state into account. As a result, personalized suggestions based on the user's mental state and preferences are insufficient, limiting the effectiveness of health management. The present invention aims to realize more personalized dietary and health management by recognizing the user's emotions in real time and adjusting the suggestions based on this.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0988] In this invention, the server includes means for receiving images of meals taken by a user, means for analyzing the images to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for managing the user's meals based on the stored information about ingredients and dishes, means for recognizing the user's emotions from facial expressions, tone of voice, and text data, and means for adjusting the content of suggestions based on the recognized emotions, thereby enabling personalized suggestions based on the user's emotional state.

[0989] A "user" is an individual who uses the system to take photos of their meals and use those photos for analysis.

[0990] The "server" is a central processing unit that receives image data sent by users and performs analysis and proposals.

[0991] A "terminal" is a device used by a user, such as a smartphone or tablet, that has the function of sending captured images to a server.

[0992] A "meal image" is a photograph of a meal taken by a user, and serves as data for identifying ingredients and dishes.

[0993] "Analysis" refers to the process performed to identify ingredients and dish information from the received food image.

[0994] "Ingredient and dish information" refers to the names of ingredients, dishes, amounts, calories, and other nutritional information identified through analysis.

[0995] "Database" refers to an information management system for storing information on identified ingredients and dishes.

[0996] "Dietary management" refers to the act of managing the calorie and nutrient intake of a user based on stored information about ingredients and dishes.

[0997] "Means for recognizing emotions" refers to technology that has the ability to analyze a user's emotions from their facial expressions, tone of voice, and text data.

[0998] "Means to adjust suggestion content" refers to a function that appropriately changes the content of meal management, repurchase, and recipe suggestions based on recognized emotions.

[0999] The invention begins with a user taking a picture of a meal using a device such as a smartphone or tablet and sending the image to a server. The device compresses the image using a library such as Python's Pillow and sends it to the server in an appropriate format (e.g., JPEG or PNG). The server then inputs the received image data into an image recognition module, which uses a convolutional neural network (CNN) and an AI model.

[1000] The image recognition module uses machine learning algorithms to identify ingredient and dish information from the submitted image. The identified information includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information. This information is stored in a database on the server. This eliminates the need for users to record their meals themselves, allowing for automated diet management.

[1001] The present invention also incorporates an emotion engine, allowing the server to recognize emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state, and if the user is feeling stressed or tired, for example, it will suggest ingredients and dishes that have a relaxing effect.

[1002] For example, a user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. When the device sends the photo to the server, the server's image recognition module analyzes it and identifies the ingredients, such as chicken, rice, lettuce, and tomato. The nutritional information for these ingredients is then stored in a database, and the user's meal record is updated. Furthermore, the emotion engine analyzes the user's emotional state and, if necessary, suggests relaxing herbal teas.

[1003] Examples of prompt sentences include the following:

[1004] Ingredient identification model

[1005] "Identify the ingredients and name of the dish based on a photo of the meal taken by the user."

[1006] Emotion Engine

[1007] "Analyze the user's emotions from facial expressions, tone of voice, and text data, and generate appropriate suggestions."

[1008] In this way, by appropriately utilizing the generative AI model and prompt sentences, personalized suggestions based on the user's condition become possible. This system enables efficient and effective dietary management for users, and also realizes health management that takes into account the user's emotional state.

[1009] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1010] Step 1: User takes and submits a photo of their meal

[1011] Specific actions and inputs

[1012] The user opens the smartphone app and takes a photo of their meal. This photo becomes the input data. Once the photo is taken, the user taps the "Send Photo" button.

[1013] Data processing and output

[1014] The device compresses the captured image and converts it to the appropriate format (e.g., JPEG or PNG). The image is compressed using the Python Pillow library and sent to the server. The compressed image data is the output.

[1015] Step 2: The server receives the image data and analyzes it.

[1016] Specific actions and inputs

[1017] The server receives the compressed image data sent from the terminal. The received image data is input data.

[1018] Data processing and output

[1019] The server's image recognition module analyzes the received image data and uses a convolutional neural network (CNN) to identify information about ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1020] Step 3: The server saves the analysis results to the database and updates the food record.

[1021] Specific actions and inputs

[1022] The analysis results, which are information on ingredients and dishes, become input data.

[1023] Data processing and output

[1024] The server stores the identified ingredients and dish information in a database. Based on this stored data, the user's food record is updated, including the total calories, protein, carbohydrates, and fat intake. The updated food record is the output.

[1025] Step 4: The server analyzes the meal record and suggests ingredients and seasonings that need to be repurchased.

[1026] Specific actions and inputs

[1027] The server receives the latest food record data as input.

[1028] Data processing and output

[1029] The server analyzes consumption over a specific period (e.g., one week) and identifies ingredients and seasonings that exceed a consumption threshold. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. This suggestion message is the output.

[1030] Step 5: The device sends a repurchase suggestion message to the user

[1031] Specific actions and inputs

[1032] The repurchase offer message sent from the server is the input data.

[1033] Data processing and output

[1034] The device sends a repurchase suggestion message to the user as a push notification. The user confirms the notification and is presented with a link to repurchase based on the suggestion. The notified suggestion message is the output.

[1035] Step 6: The server analyzes the food record and preference data and suggests recipes

[1036] Specific actions and inputs

[1037] The user's food records and preference data are the input data.

[1038] Data processing and output

[1039] The server analyzes this data and automatically generates a recipe that takes nutritional balance into consideration. It creates a recipe suggestion message that includes the necessary ingredients, quantities, and cooking steps. The generated recipe is the output.

[1040] Step 7: The device notifies the user of the recipe suggestion message

[1041] Specific actions and inputs

[1042] The recipe suggestion message sent from the server is the input data.

[1043] Data processing and output

[1044] The device sends a recipe suggestion message to the user as a push notification. The user can check the notification and view the recipe details. The notified recipe suggestion message is the output.

[1045] Step 8: The server recognizes the user's emotions using the emotion engine and adjusts the suggestions.

[1046] Specific actions and inputs

[1047] The input data includes the user's facial expression, tone of voice, and text data.

[1048] Data processing and output

[1049] The server uses an emotion engine to analyze the user's emotions from these data in real time. Based on the recognized emotions, the server adjusts the recommendations, such as ingredients and dishes that have a relaxing effect. The adjusted recommendation message is output.

[1050] Step 9: The device notifies the user with a sentiment-based suggestion message

[1051] Specific actions and inputs

[1052] The input data is a suggestion message based on the emotion sent from the server.

[1053] Data processing and output

[1054] The device sends a sentiment-based suggestion message to the user as a push notification. The user acknowledges the notification and is provided with guidance to take appropriate action. The notified suggestion message is the output.

[1055] Through the above steps, the system of the present invention realizes the user's dietary management, repurchase suggestions, recipe suggestions based on nutritional balance, and personalized health management according to the user's emotional state.

[1056] (Application example 2)

[1057] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1058] Conventional dietary management systems are limited to managing users' dietary records and nutritional balance, and have the problem of not being able to provide dietary suggestions based on the user's emotional state. This makes it difficult to manage health care that takes the user's emotional state into account, and there are issues with not being able to provide personalized suggestions. Furthermore, there is also the problem that repurchase suggestions and online ordering are not automated, making it difficult for users to efficiently replenish their food supplies.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1060] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified information on ingredients and dishes in a database, a means for managing the user's diet based on the stored information on ingredients and dishes, a means for recognizing emotions from the user's facial expressions, tone of voice, and text data, and a means for adjusting suggestions based on the recognized emotions. This allows for more refined diet management for the user and makes suggestions tailored to their emotions. Furthermore, it enables automated repurchase suggestions and online ordering, improving user convenience.

[1061] "Means for receiving images of meals taken by the user" is a function for collecting image data of meals taken using a device such as a smartphone or digital camera and sending it to a server or cloud.

[1062] The "means for analyzing the image and identifying information about ingredients and dishes" is a function that uses machine learning algorithms and image recognition technology to identify the types of ingredients and dishes contained in the image based on the received image data.

[1063] The "means for saving information on the identified ingredients and dishes in a database" is a function for recording and accumulating the analyzed ingredients and dish types and related nutritional information in a database.

[1064] "Means for managing the user's diet based on stored information on ingredients and dishes" refers to a function that uses information stored in the database to manage the user's dietary history, nutritional intake, calories, etc., and to suggest appropriate meals and manage health.

[1065] "Means for recognizing emotions from a user's facial expression, tone of voice, and text data" is a function for analyzing emotions from a user's facial expression data, tone of voice, and text content, and identifying that state.

[1066] The "means for adjusting the content of suggestions based on the recognized emotion" is a function for using the recognized emotion information to appropriately change the content of suggestions for meals and ingredients to the user, and providing services that correspond to the user's emotional state.

[1067] The "means for suggesting ingredients and seasonings that need to be repurchased" is a function that identifies ingredients and seasonings that are in short supply based on the user's meal record and information on ingredients consumed, and suggests to the user that they be repurchased.

[1068] The "means for placing an online order based on the suggestions" is a function that provides a link or interface for the user to place an order for the suggested ingredients or seasonings at an online shop.

[1069] The "means for providing links related to delivery services" is a function that provides users with links for ordering the suggested dishes and ingredients through delivery services.

[1070] In one embodiment of the present invention, a system is used that receives images of food taken by a user, identifies ingredients and dishes, and makes suggestions based on the user's emotional state. This system is realized using the user's smartphone, a server, a database, an emotion recognition engine, etc.

[1071] 1. Meal photo recognition and management

[1072] User

[1073] Users use a smartphone app to take a photo of their meal and tap the "send" button.

[1074] Terminal

[1075] The device compresses the captured image and sends it to the server, where the image data is sent in the appropriate format (e.g., JPEG or PNG).

[1076] server

[1077] The server inputs the received image data into an image recognition module and uses machine learning algorithms (e.g., CNN) to identify the ingredients and dishes in the image. The identified ingredients and dishes are stored in a database. This information includes the names of the ingredients, the names of the dishes, the serving sizes, calories, and other nutritional information. The stored information is also used to update the user's food log and manage calorie and macronutrient intake.

[1078] Specific examples

[1079] For example, a user takes a photo of their lunch of "teriyaki chicken," "rice," and "salad," and the device sends the photo to the server. The server analyzes the image, identifies the chicken, rice, lettuce, and tomato, and stores their nutritional information in the user's database. This data is used to manage the user's nutritional intake.

[1080] 2. Suggestions for repurchasing ingredients and seasonings

[1081] server

[1082] The server periodically analyzes the user's food log and checks the food ingredient data consumed. This analysis includes the amount consumed over a specific period (e.g., one week). If an ingredient or seasoning exceeds a threshold, it determines that it needs to be repurchased. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. The message includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1083] Terminal

[1084] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[1085] User

[1086] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1087] Specific examples

[1088] The server analyzes the recent food log and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[1089] 3. Recipe suggestions

[1090] server

[1091] The server analyzes the user's food records and preference data, including the user's favorite ingredients and nutritional balance data. It automatically generates recipes that take nutritional balance into consideration and selects recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences. It then creates a message to notify the user of the generated recipe. The message contains detailed information such as the recipe name, required ingredients, quantities, and cooking instructions.

[1092] Terminal

[1093] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1094] User

[1095] The user confirms the notification and views the details of the suggested recipe. The user then prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1096] Specific examples

[1097] The server analyzes the user's recent food log and detects that they are iron deficient. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user checks the notification and cooks the dish according to the recipe.

[1098] 4. Incorporating an Emotional Engine

[1099] server

[1100] The server uses an emotion engine that recognizes the user's emotions from their facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state and adjust the recommendations based on the recognized emotion. For example, if the user is feeling stressed, the server will suggest ingredients and dishes that have a relaxing effect.

[1101] Terminal

[1102] The terminal receives the emotion-based suggestion message from the server and notifies the user.

[1103] User

[1104] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[1105] Specific examples

[1106] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[1107] Prompt Sentence Examples

[1108] "Analyze the user's food photos and identify the food and ingredients. Also, determine the user's emotions from their facial expressions and tone of voice, and suggest ingredients and recipes based on that."

[1109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1110] Step 1:

[1111] A user takes a photo of their meal using their smartphone and taps the "send" button.

[1112] Input: Image file of the food you have taken

[1113] Output: Food image data sent to the server

[1114] Specific operation: Take a photo of your meal using the smartphone's camera app, and send the image to the server using the "Send" button in the application.

[1115] Step 2:

[1116] The device compresses the images it takes and sends them to the server.

[1117] Input: Image file of the photographed meal

[1118] Output: Compressed food image data (JPEG or PNG format)

[1119] Specific operation: The device compresses the image file into the appropriate format and sends it to the specified server address.

[1120] Step 3:

[1121] The image data received by the server is input into an image recognition module to identify the ingredients and the dish.

[1122] Input: Compressed food image data

[1123] Output: Identified ingredients and dish information (e.g., ingredient name, dish name)

[1124] Specific operation: The received image data is input into a machine learning algorithm (e.g., a CNN model) to identify the ingredients and dishes in the image.

[1125] Step 4:

[1126] The server stores the identified ingredients and dish information in a database.

[1127] Input: Identified ingredients and dish information

[1128] Output: Updated food record in the database

[1129] Specific behavior: Writes the identified ingredients and dish data to a database and updates the user's meal record.

[1130] Step 5:

[1131] The server recognizes the user's emotions from their facial expressions, tone of voice, and text data.

[1132] Input: User's facial expression data, voice data, text data

[1133] Output: Perceived emotional state (e.g., stress, happiness)

[1134] Specific operation: Facial expressions and tone of voice collected by the smartphone's camera and microphone are input into an emotion recognition engine to identify the user's emotional state.

[1135] Step 6:

[1136] The server adjusts the suggestions based on the perceived sentiment.

[1137] Input: Perceived emotional state

[1138] Output: Tailored recommendations (e.g., ingredients and dishes with relaxing effects)

[1139] What it does: It applies algorithms that change the food and recipe suggestions provided to users based on their emotional state.

[1140] Step 7:

[1141] The server notifies the user of suggested recipes and repurchase suggestions.

[1142] Input: Adjusted recommendations, ingredients and seasonings that need to be repurchased

[1143] Output: A notification message displayed on the user's terminal.

[1144] Specific behavior: A suggestion message is sent to the user's smartphone as a push notification, and the user receives the notification.

[1145] Step 8:

[1146] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions.

[1147] Input: Notification content (recipe information, repurchase link)

[1148] Output: Order completed at online store

[1149] Specific behavior: The user taps the notification to be taken to the linked online shop, where they can add the suggested ingredients and seasonings to their cart and place an order.

[1150] Step 9:

[1151] The server generates a message suggesting ingredients and seasonings that need to be repurchased and sends it to the user.

[1152] Input: Food log and food consumption data

[1153] Output: Suggestion message (list of missing ingredients and seasonings)

[1154] Specific operation: Analyzes food records, identifies ingredients and seasonings consumed frequently, and generates suggestion messages based on that information and sends them to the user.

[1155] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1157] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1158] [Third embodiment]

[1159] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1160] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1162] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1166] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1167] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1168] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1169] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1170] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1171] The present invention relates to a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. An embodiment of this system will be described in detail below.

[1172] Meal photo recognition and management

[1173] User

[1174] The user takes a photo of the meal and launches the application, which saves the image of the meal on the user's device.

[1175] Terminal

[1176] The device sends the images taken by the user to the server, where the image data is compressed in an appropriate format to ensure efficient data transfer.

[1177] server

[1178] The server analyzes the images received from the device using machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify ingredients and dishes in the images.

[1179] The identified ingredients and dish information is stored in a database, which includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1180] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[1181] Specific examples

[1182] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[1183] Repurchase suggestions for ingredients and seasonings

[1184] server

[1185] The server periodically analyzes the user's food log and consumed food ingredient data, for example, to identify ingredients or condiments consumed in amounts above a certain threshold.

[1186] The server lists ingredients and seasonings that need to be repurchased and generates a suggestion message to notify the user at an appropriate time.

[1187] Terminal

[1188] The device then sends a repurchase suggestion message to the user, which includes the suggested ingredients and seasonings and a link to an online shop for repurchasing.

[1189] User

[1190] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[1191] Specific examples

[1192] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[1193] Recipe suggestions

[1194] server

[1195] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, including the user's preferred ingredients and past meal history.

[1196] The server creates a message to notify the user of the created recipe, which includes detailed information about the ingredients, quantities, cooking steps, etc.

[1197] Terminal

[1198] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[1199] User

[1200] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to follow a nutritionally balanced diet.

[1201] Specific examples

[1202] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[1203] conclusion

[1204] In this way, the system analyzes photos of meals taken by the user to manage their diet, suggests repurchases of consumed ingredients and seasonings, and provides nutritionally balanced recipes. This allows users to efficiently manage their health while reducing the effort required for managing and purchasing ingredients.

[1205] The processing flow will be explained below.

[1206] Meal photo recognition and management

[1207] Step 1:

[1208] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[1209] Step 2:

[1210] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[1211] Step 3:

[1212] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[1213] Step 4:

[1214] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1215] Step 5:

[1216] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[1217] Repurchase suggestions for ingredients and seasonings

[1218] Step 1:

[1219] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[1220] Step 2:

[1221] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[1222] Step 3:

[1223] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1224] Step 4:

[1225] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the ingredients and seasonings needed and a link to repurchase them.

[1226] Step 5:

[1227] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1228] Recipe suggestions

[1229] Step 1:

[1230] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[1231] Step 2:

[1232] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[1233] Step 3:

[1234] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[1235] Step 4:

[1236] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1237] Step 5:

[1238] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1239] summary

[1240] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes, allowing them to efficiently manage their daily health.

[1241] Example 1

[1242] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1243] When users manage their daily diet, it is time-consuming to understand the details of ingredients and dishes and update records based on accurate nutritional information. Furthermore, knowing when to repurchase consumed ingredients and seasonings and suggesting recipes that take nutritional balance into consideration are also significant burdens for individual users. This makes ongoing health management difficult, creating a challenge.

[1244] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1245] In this invention, the server includes means for analyzing images of meals taken by the user to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for updating the user's meal record based on the stored information, means for analyzing the user's meal record and consumed ingredients and suggesting ingredients and seasonings that need to be repurchased, and means for suggesting recipes that take the user's nutritional balance into consideration based on the user's meal record and preference data. This allows the user to continuously and easily manage their diet, appropriately repurchase consumed ingredients and seasonings, and prepare meals that take nutritional balance into consideration.

[1246] A "user" is someone who uses the system to access functions such as dietary management, repurchasing ingredients, and recipe suggestions.

[1247] A "terminal" refers to an electronic device such as a smartphone, tablet, or PC used by a user to take photos of meals, send images, receive notifications, and so on.

[1248] A "server" is a computer system that analyzes image data sent from a terminal, stores the results in a database, and makes various suggestions to the user.

[1249] "Image data" refers to digital information of a photograph of a meal taken by a user, which is analyzed by the server.

[1250] "Machine learning algorithms" refer to artificial intelligence techniques used by the server to analyze image data, including convolutional neural networks (CNNs).

[1251] "Ingredients" are the individual food components in a meal that contain nutritional information.

[1252] A "dish" is a meal prepared according to a specific recipe and consisting of a combination of ingredients.

[1253] A "database" is an electronic recording device that stores information on ingredients and dishes, user meal records, preference data, and so on.

[1254] A "repurchase suggestion message" is a notification message sent by the server after analyzing the user's consumption data regarding ingredients or seasonings that need to be repurchased.

[1255] "Nutritional balance" refers to the distribution of each nutrient in the user's diet, and indicates whether it is balanced for health management.

[1256] "Recipe suggestions" refers to the server providing users with cooking methods that take nutritional balance into consideration based on the user's meal records and preference data.

[1257] "Notification" refers to the means by which a device communicates information such as repurchase suggestions or recipe suggestions to a user, and includes push notifications.

[1258] This invention relates to a system that analyzes images of meals taken by users, identifies ingredients and dishes, and performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. The system of the present invention is configured to achieve the following three main functions:

[1259] 1. Meal photo recognition and management

[1260] User

[1261] To record the contents of their meals, users take photos of their meals using the device's camera. The images are then saved directly to the device.

[1262] Terminal

[1263] The device has the ability to compress the image data captured by the user into an appropriate format (e.g., JPEG or PNG) and send it to the server. Specifically, it uses a compression algorithm to ensure efficient data transfer.

[1264] server

[1265] The server receives the image data sent from the device and analyzes it using a machine learning algorithm (e.g., convolutional neural network: CNN) to identify the ingredients and dishes.

[1266] The server stores the identified ingredients and dish information (e.g., ingredient name, dish name, portion size, calories, nutritional information) in a database.

[1267] Based on the saved information, it has the function of updating the user's food record and calculating and saving the total calories and nutritional components ingested.

[1268] Specific examples

[1269] A user takes a photo of their lunch: teriyaki chicken, rice, and salad. The device compresses the photo into JPEG format and sends it to the server. The server performs image analysis, identifies the ingredients (chicken, rice, lettuce, and tomato), and stores their nutritional information in a database. Finally, the user's food record is updated, enabling accurate nutritional management.

[1270] 2. Suggestions for repurchasing ingredients and seasonings

[1271] server

[1272] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[1273] The server generates a repurchase suggestion message regarding ingredients and seasonings that need to be repurchased and transmits it to the user's terminal.

[1274] Terminal

[1275] The device displays the repurchase suggestion message sent from the server to the user as a push notification, which includes the name of the suggested ingredient or seasoning and a link to the online shop for repurchase.

[1276] User

[1277] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[1278] Specific examples

[1279] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps a link to order soy sauce from an online shop.

[1280] 3. Recipe suggestions

[1281] server

[1282] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, and suggests appropriate recipes based on the user's preferred ingredients and past meal history.

[1283] Create a message to notify the user of the generated recipe, which includes detailed information such as ingredients, quantities, and cooking steps.

[1284] Terminal

[1285] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[1286] User

[1287] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to eat a nutritionally balanced diet.

[1288] Specific examples

[1289] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their iron.

[1290] conclusion

[1291] This system analyzes images of meals taken by users and provides dietary management, suggestions for repurchasing ingredients, and recipe suggestions based on nutritional balance, allowing users to efficiently manage their health and reducing the effort required for managing and purchasing ingredients, making ongoing health management easier.

[1292] Prompt Sentence Examples

[1293] "Please explain in natural language the program for a system that takes a photo of a meal using an app, sends it to a server, identifies ingredients and dishes from the analysis results, and updates the food record based on the nutritional information. Please include specific examples."

[1294] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1295] Step 1:

[1296] The user takes a photo of their meal, and the image is saved in the device's internal storage.

[1297] Input: A user-taken image of a meal.

[1298] How it works: You take a photo using the camera app on your smartphone or tablet, which creates an image file.

[1299] Output: Image file saved in the device's internal storage.

[1300] Step 2:

[1301] The device compresses the image data and sends it to the server. The device uses a compression format such as JPEG or PNG.

[1302] Input: An image file stored in the device's internal storage.

[1303] How it works: An image compression algorithm is run on the device to generate a compressed image file, which is then sent over the internet to a server.

[1304] Output: Compressed image data sent to the server.

[1305] Step 3:

[1306] The server analyzes the received image data and uses machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify the ingredients and dishes in the image.

[1307] Input: Compressed image data sent from the terminal.

[1308] How it works: Machine learning algorithms run on the server to analyze the image, identify ingredients and dishes, and extract information about them.

[1309] Output: Information on identified ingredients and dishes (ingredients, dish name, portion size, calories, nutritional information, etc.).

[1310] Step 4:

[1311] The server stores the identified ingredients and dish information in a database.

[1312] Input: Ingredient and dish information identified by the analysis.

[1313] Operation: The analysis results are written and saved in the server's database system.

[1314] Output: Ingredient and dish information stored in a database.

[1315] Step 5:

[1316] The server updates the user's food record based on the stored information, calculates the total calories and nutritional information, and updates the database.

[1317] Input: Identified ingredients and dish information.

[1318] How it works: The server runs a nutrition calculation algorithm that adds calorie and nutrient totals to each user's food log.

[1319] Output: Updated user's meal log.

[1320] Step 6:

[1321] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[1322] Input: User's food log and consumed ingredients data.

[1323] How it works: The server runs a data analysis algorithm to identify ingredients and seasonings that are being consumed most frequently.

[1324] Output: A list of ingredients and seasonings that need to be repurchased.

[1325] Step 7:

[1326] The server generates a repurchase offer message and sends it to the user's terminal.

[1327] Input: A list of ingredients and seasonings that need to be repurchased.

[1328] Behavior: The server generates a proposal message and sends it to the device as a push notification.

[1329] Output: Repurchase offer message sent to the user's device.

[1330] Step 8:

[1331] The user checks the repurchase suggestion message and orders ingredients and seasonings from the online shop as needed.

[1332] Input: Repurchase suggestion message.

[1333] Action: The user sees the notification and taps the link to go to the online store and order the ingredients and seasonings they need.

[1334] Output: The ingredients and seasonings for the completed order.

[1335] Step 9:

[1336] The server analyzes the user's food records and preference data and generates recipes that take nutritional balance into consideration.

[1337] Input: User's food log and preference data.

[1338] How it works: Data analysis is performed on the server to generate nutritionally balanced recipes that are optimal for the user.

[1339] Output: Nutritionally balanced recipes.

[1340] Step 10:

[1341] The server notifies the user of the generated recipe.

[1342] Input: The generated recipe.

[1343] Operation: The server generates a recipe notification message and sends it to the device as a push notification.

[1344] Output: Recipe notification sent to the user's device.

[1345] Step 11:

[1346] The user checks the recipe notification and cooks using the suggested recipe as a reference.

[1347] Enter: recipe notification.

[1348] What happens: The user opens the notification, prepares the ingredients, and follows the recipe displayed to cook the food.

[1349] Output: A nutritionally balanced diet.

[1350] The above is the flow of processing of the program of this system.

[1351] (Application example 1)

[1352] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1353] In modern society, it is important for users to manage their health on a daily basis, but it is difficult to efficiently manage their diet and adjust nutritional balance in a busy lifestyle. There is also a lack of comprehensive systems that support overall dietary habits, such as managing food purchases and suggesting new recipes. Therefore, there is a need for a system that can identify information about ingredients and dishes based on photos of meals taken by users, and then use that information to manage their diet, suggest repurchasing ingredients, suggest recipes that take nutritional balance into consideration, and even deliver health-related content.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1355] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified ingredient and dish information in a database, a means for managing the user's diet based on the stored ingredient and dish information, and a means for delivering health-related content. This allows users to automate their daily diet management and live a nutritionally balanced and healthy lifestyle. Furthermore, the server includes a means for suggesting ingredients and seasonings that need to be repurchased, a means for suggesting recipes based on the user's preference data, and a means for suggesting personalized health content using a generative AI model, enabling efficient comprehensive health management.

[1356] "User" refers to any individual or organization that uses this system.

[1357] "Meal image" refers to a digital image taken by a user that visually records the contents of a meal.

[1358] "Analysis" refers to the process of identifying ingredients and dish information from received food images, primarily using machine learning algorithms.

[1359] "Ingredient and dish information" is data indicating the components of a meal identified through image analysis, and includes the names of ingredients, dish names, quantities, nutritional components, etc.

[1360] "Database" refers to a digital storage system that organizes and stores information about identified ingredients and dishes so that it can be retrieved and updated later.

[1361] "Diet management" refers to the activity of tracking a user's diet and monitoring and adjusting the balance of calories and nutrients ingested.

[1362] "Health-related content" refers to digital content such as videos and articles that provide customized information and advice based on a user's health and diet.

[1363] "Repurchase suggestions" refers to notifications or messages that identify ingredients or seasonings that are missing based on the user's meal records and suggest purchasing them.

[1364] "Online ordering" refers to the process by which a user purchases suggested ingredients and seasonings through an online shopping system.

[1365] "Preference data" refers to personal data about ingredients or dishes selected by the user in the past, or the user's preferences.

[1366] "Recipe suggestion" refers to the activity of informing users of nutritionally balanced recipes based on their food records and preference data.

[1367] "Generative AI model" refers to the artificial intelligence algorithm used to analyze a user's eating habits and suggest personalized health content.

[1368] "Personalized health content" refers to health-related information and advice that is customized for a particular user based on the user's individual food records and nutritional status.

[1369] This invention provides a system for managing meals by analyzing images of meals taken by users and identifying information about ingredients and dishes. This system aims to provide comprehensive health management, including suggestions for repurchasing ingredients, recipe suggestions that take nutritional balance into consideration, and the distribution of health-related content.

[1370] 1. System Overview

[1371] The system consists of three main parts: user terminals, servers, and databases.

[1372] 2. User Device

[1373] The user device is a smartphone or tablet with a camera. The user takes a photo of the meal and launches a dedicated application. This application has the function of acquiring the captured image, compressing it in an appropriate format, and sending it to the server.

[1374] 3. Server

[1375] The server has the following functions:

[1376] Image analysis function

[1377] The server analyzes the received meal images using a convolutional neural network (CNN) to identify the ingredients and dishes in the image. The analysis results include the names of the ingredients, the dish, the serving size, calories, and other nutritional information.

[1378] Database Management Functions

[1379] The identified ingredients and dish information are stored in a database that holds the user's meal record, preference data, and data on ingredients consumed.

[1380] Meal management function

[1381] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[1382] Repurchase suggestion function

[1383] The server analyzes the user's meal log and identifies ingredients and seasonings that need to be repurchased. Once identified, the server notifies the user and provides a link to encourage online ordering.

[1384] Recipe suggestion function

[1385] The server analyzes the user's food records and preference data, and generates recipes that take into account the user's nutritional balance. The generated recipes are then notified to the user.

[1386] Health-related content distribution function

[1387] Using a generative AI model, it analyzes users' eating habits and suggests personalized health-related content, including videos and articles related to staying healthy.

[1388] 4. Specific Examples

[1389] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to a server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is stored in the user's database. The server updates the user's food record based on the nutritional information of these ingredients, providing appropriate nutritional management. The server also identifies ingredients that are in short supply and generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" The recipe suggestion function then sends the user a notification saying, "You are low on iron. We suggest a recipe using spinach and beef." The generative AI model also analyzes the user's eating habits and delivers "advice videos for increasing specific nutrients."

[1390] 5. Examples of Prompt Statements

[1391] "Based on photos of meals taken by users with their smartphones, please use a CNN model to analyze what ingredients and dishes are included, and upload the results to a database on the cloud. Also, please make sure that dietary management and recipe suggestions are performed based on the user's nutritional balance."

[1392] This system enables individual users to lead a nutritionally balanced and healthy life while automating their daily dietary management, and also enables efficient comprehensive health management.

[1393] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1394] Step 1:

[1395] The user takes a photo of the meal.

[1396] Input: Food image

[1397] Output: Raw image files saved on the device

[1398] Specific operation: The user takes a photo of the meal using the camera app on their smartphone and launches a dedicated application.

[1399] Step 2:

[1400] The device sends the captured image to the server.

[1401] Input: Image file saved on the device

[1402] Output: Compressed image data sent to the server

[1403] Specific operation: The application compresses the image file in the appropriate format and sends it to the server using a data transfer protocol (e.g., HTTP).

[1404] Step 3:

[1405] The server analyzes the received image.

[1406] Input: Compressed image data sent

[1407] Output: Analysis results (information on identified ingredients and dishes)

[1408] How it works: The server uses a convolutional neural network (CNN) to analyze the image and identify ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1409] Step 4:

[1410] The server stores the identified ingredients and dish information in a database.

[1411] Input: Analysis results (identified ingredients and dish information)

[1412] Output: Ingredient and dish information stored in a database

[1413] Specific operation: The server executes queries to store the analysis results in an SQL or NoSQL database.

[1414] Step 5:

[1415] The server manages dietary management based on the stored information.

[1416] Input: Ingredient and dish information stored in a database

[1417] Output: Data required for dietary management, updated user diet record

[1418] Specific operation: The server retrieves the stored information, calculates the total calories and nutritional intake, and updates the user's food record.

[1419] Step 6:

[1420] The server analyzes the user's meal log and the ingredients consumed.

[1421] Input: User's food records and consumed ingredients data stored in a database

[1422] Output: List of ingredients and seasonings that need to be repurchased

[1423] Specific operation: The server uses an algorithm to analyze the user's food records and identify ingredients and seasonings whose consumption exceeds a certain threshold.

[1424] Step 7:

[1425] The server generates a repurchase offer message and sends it to the terminal.

[1426] Input: List of ingredients and seasonings that need to be repurchased

[1427] Output: Repurchase suggestion message

[1428] Specific operation: The server generates a proposal message and sends it to the user's device via push notification or email.

[1429] Step 8:

[1430] The server analyzes the user's food records and preference data.

[1431] Input: Ingredient and dish information stored in the database, user preference data

[1432] Output: Generate recipes that take nutritional balance into consideration

[1433] Specific operation: The server generates nutritionally balanced recipes based on the user's preference data and food records, and uses a generative AI model to suggest personalized health content.

[1434] Step 9:

[1435] The device will notify you of suggested recipes and health-related content.

[1436] Input: Generated recipes and personalized health content

[1437] Output: Notifications and application screen display

[1438] Specific behavior: The device displays the received notification and allows the user to view recipes and health-related content.

[1439] These processing steps enable users to efficiently manage their diet and lead a nutritionally balanced and healthy life. Furthermore, the generative AI model provides personalized health content, allowing users to receive advice based on their individual health conditions.

[1440] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1441] The present invention is a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes that take nutritional balance into consideration based on this information. By further combining this with an emotion engine, the system makes suggestions based on the user's emotions. An embodiment of this system will be described in detail below.

[1442] Meal photo recognition and management

[1443] User

[1444] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[1445] Terminal

[1446] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[1447] server

[1448] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[1449] The identified ingredients and dish information is stored in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1450] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[1451] Specific examples

[1452] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[1453] Repurchase suggestions for ingredients and seasonings

[1454] server

[1455] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[1456] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[1457] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1458] Terminal

[1459] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[1460] User

[1461] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1462] Specific examples

[1463] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[1464] Recipe suggestions

[1465] server

[1466] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[1467] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[1468] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[1469] Terminal

[1470] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1471] User

[1472] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1473] Specific examples

[1474] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[1475] Incorporating an emotion engine

[1476] server

[1477] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[1478] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[1479] Terminal

[1480] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[1481] User

[1482] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[1483] Specific examples

[1484] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[1485] conclusion

[1486] This invention provides a unique function that combines an emotion engine with a system that analyzes images of meals taken by users to manage their diet, suggest repurchasing ingredients, and suggest recipes based on nutritional balance. This enables health management according to the user's emotional state and makes more personalized suggestions.

[1487] The processing flow will be explained below.

[1488] Meal photo recognition and management

[1489] Step 1:

[1490] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[1491] Step 2:

[1492] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[1493] Step 3:

[1494] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[1495] Step 4:

[1496] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1497] Step 5:

[1498] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[1499] Repurchase suggestions for ingredients and seasonings

[1500] Step 1:

[1501] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[1502] Step 2:

[1503] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[1504] Step 3:

[1505] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1506] Step 4:

[1507] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the suggested ingredients and seasonings and a link to repurchase them.

[1508] Step 5:

[1509] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1510] Recipe suggestions

[1511] Step 1:

[1512] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[1513] Step 2:

[1514] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[1515] Step 3:

[1516] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[1517] Step 4:

[1518] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1519] Step 5:

[1520] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1521] Incorporating an emotion engine

[1522] Step 1:

[1523] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[1524] Step 2:

[1525] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[1526] Step 3:

[1527] The server creates emotion-based suggestion messages, which include suggestions for ingredients and dishes based on the user's emotional state.

[1528] Step 4:

[1529] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[1530] Step 5:

[1531] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[1532] Specific examples

[1533] Example 1: Food photo recognition

[1534] A user takes a photo of "toast and coffee" for breakfast. The device sends the photo to the server, which analyzes the photo and identifies the toast and coffee. This nutritional information is stored in a database and the user's food record is updated.

[1535] Example 2: Repurchase offer

[1536] The server analyzes the user's food log and detects that milk consumption is increasing. The server generates a repurchase suggestion message saying "Milk stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order milk from the online shop.

[1537] Example 3: Recipe suggestions

[1538] The server analyzes the user's food records and detects a vitamin D deficiency. The server generates a nutritionally balanced recipe using fish and spinach and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their vitamin D.

[1539] Example 4: Emotion engine suggestions

[1540] While a user is using a smartphone app, the emotion engine recognizes that the user is tired. The server generates a suggestion message saying, "How about some relaxing herbal tea?" and sends it to the device. The user can check the notification and prepare the suggested herbal tea to relax.

[1541] conclusion

[1542] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes. Furthermore, by incorporating an emotion engine, suggestions can be made based on the user's emotional state, enabling even more personalized support.

[1543] Example 2

[1544] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1545] Conventional dietary management systems analyze nutritional information and suggest repurchasing ingredients, but are unable to provide suggestions that take the user's emotional state into account. As a result, personalized suggestions based on the user's mental state and preferences are insufficient, limiting the effectiveness of health management. The present invention aims to realize more personalized dietary and health management by recognizing the user's emotions in real time and adjusting the suggestions based on this.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1547] In this invention, the server includes means for receiving images of meals taken by a user, means for analyzing the images to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for managing the user's meals based on the stored information about ingredients and dishes, means for recognizing the user's emotions from facial expressions, tone of voice, and text data, and means for adjusting the content of suggestions based on the recognized emotions, thereby enabling personalized suggestions based on the user's emotional state.

[1548] A "user" is an individual who uses the system to take photos of their meals and use those photos for analysis.

[1549] The "server" is a central processing unit that receives image data sent by users and performs analysis and proposals.

[1550] A "terminal" is a device used by a user, such as a smartphone or tablet, that has the function of sending captured images to a server.

[1551] A "meal image" is a photograph of a meal taken by a user, and serves as data for identifying ingredients and dishes.

[1552] "Analysis" refers to the process performed to identify ingredients and dish information from the received food image.

[1553] "Ingredient and dish information" refers to the names of ingredients, dishes, amounts, calories, and other nutritional information identified through analysis.

[1554] "Database" refers to an information management system for storing information on identified ingredients and dishes.

[1555] "Dietary management" refers to the act of managing the calorie and nutrient intake of a user based on stored information about ingredients and dishes.

[1556] "Means for recognizing emotions" refers to technology that has the ability to analyze a user's emotions from their facial expressions, tone of voice, and text data.

[1557] "Means to adjust suggestion content" refers to a function that appropriately changes the content of meal management, repurchase, and recipe suggestions based on recognized emotions.

[1558] The invention begins with a user taking a picture of a meal using a device such as a smartphone or tablet and sending the image to a server. The device compresses the image using a library such as Python's Pillow and sends it to the server in an appropriate format (e.g., JPEG or PNG). The server then inputs the received image data into an image recognition module, which uses a convolutional neural network (CNN) and an AI model.

[1559] The image recognition module uses machine learning algorithms to identify ingredient and dish information from the submitted image. The identified information includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information. This information is stored in a database on the server. This eliminates the need for users to record their meals themselves, allowing for automated diet management.

[1560] The present invention also incorporates an emotion engine, allowing the server to recognize emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state, and if the user is feeling stressed or tired, for example, it will suggest ingredients and dishes that have a relaxing effect.

[1561] For example, a user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. When the device sends the photo to the server, the server's image recognition module analyzes it and identifies the ingredients, such as chicken, rice, lettuce, and tomato. The nutritional information for these ingredients is then stored in a database, and the user's meal record is updated. Furthermore, the emotion engine analyzes the user's emotional state and, if necessary, suggests relaxing herbal teas.

[1562] Examples of prompt sentences include the following:

[1563] Ingredient identification model

[1564] "Identify the ingredients and name of the dish based on a photo of the meal taken by the user."

[1565] Emotion Engine

[1566] "Analyze the user's emotions from facial expressions, tone of voice, and text data, and generate appropriate suggestions."

[1567] In this way, by appropriately utilizing the generative AI model and prompt sentences, personalized suggestions based on the user's condition become possible. This system enables efficient and effective dietary management for users, and also realizes health management that takes into account the user's emotional state.

[1568] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1569] Step 1: User takes and submits a photo of their meal

[1570] Specific actions and inputs

[1571] The user opens the smartphone app and takes a photo of their meal. This photo becomes the input data. Once the photo is taken, the user taps the "Send Photo" button.

[1572] Data processing and output

[1573] The device compresses the captured image and converts it to the appropriate format (e.g., JPEG or PNG). The image is compressed using the Python Pillow library and sent to the server. The compressed image data is the output.

[1574] Step 2: The server receives the image data and analyzes it.

[1575] Specific actions and inputs

[1576] The server receives the compressed image data sent from the terminal. The received image data is input data.

[1577] Data processing and output

[1578] The server's image recognition module analyzes the received image data and uses a convolutional neural network (CNN) to identify information about ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1579] Step 3: The server saves the analysis results to the database and updates the food record.

[1580] Specific actions and inputs

[1581] The analysis results, which are information on ingredients and dishes, become input data.

[1582] Data processing and output

[1583] The server stores the identified ingredients and dish information in a database. Based on this stored data, the user's food record is updated, including the total calories, protein, carbohydrates, and fat intake. The updated food record is the output.

[1584] Step 4: The server analyzes the meal record and suggests ingredients and seasonings that need to be repurchased.

[1585] Specific actions and inputs

[1586] The server receives the latest food record data as input.

[1587] Data processing and output

[1588] The server analyzes consumption over a specific period (e.g., one week) and identifies ingredients and seasonings that exceed a consumption threshold. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. This suggestion message is the output.

[1589] Step 5: The device sends a repurchase suggestion message to the user

[1590] Specific actions and inputs

[1591] The repurchase offer message sent from the server is the input data.

[1592] Data processing and output

[1593] The device sends a repurchase suggestion message to the user as a push notification. The user confirms the notification and is presented with a link to repurchase based on the suggestion. The notified suggestion message is the output.

[1594] Step 6: The server analyzes the food record and preference data and suggests recipes

[1595] Specific actions and inputs

[1596] The user's food records and preference data are the input data.

[1597] Data processing and output

[1598] The server analyzes this data and automatically generates a recipe that takes nutritional balance into consideration. It creates a recipe suggestion message that includes the necessary ingredients, quantities, and cooking steps. The generated recipe is the output.

[1599] Step 7: The device notifies the user of the recipe suggestion message

[1600] Specific actions and inputs

[1601] The recipe suggestion message sent from the server is the input data.

[1602] Data processing and output

[1603] The device sends a recipe suggestion message to the user as a push notification. The user can check the notification and view the recipe details. The notified recipe suggestion message is the output.

[1604] Step 8: The server recognizes the user's emotions using the emotion engine and adjusts the suggestions.

[1605] Specific actions and inputs

[1606] The input data includes the user's facial expression, tone of voice, and text data.

[1607] Data processing and output

[1608] The server uses an emotion engine to analyze the user's emotions from these data in real time. Based on the recognized emotions, the server adjusts the recommendations, such as ingredients and dishes that have a relaxing effect. The adjusted recommendation message is output.

[1609] Step 9: The device notifies the user with a sentiment-based suggestion message

[1610] Specific actions and inputs

[1611] The input data is a suggestion message based on the emotion sent from the server.

[1612] Data processing and output

[1613] The device sends a sentiment-based suggestion message to the user as a push notification. The user acknowledges the notification and is provided with guidance to take appropriate action. The notified suggestion message is the output.

[1614] Through the above steps, the system of the present invention realizes the user's dietary management, repurchase suggestions, recipe suggestions based on nutritional balance, and personalized health management according to the user's emotional state.

[1615] (Application example 2)

[1616] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1617] Conventional dietary management systems are limited to managing users' dietary records and nutritional balance, and have the problem of not being able to provide dietary suggestions based on the user's emotional state. This makes it difficult to manage health care that takes the user's emotional state into account, and there are issues with not being able to provide personalized suggestions. Furthermore, there is also the problem that repurchase suggestions and online ordering are not automated, making it difficult for users to efficiently replenish their food supplies.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1619] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified information on ingredients and dishes in a database, a means for managing the user's diet based on the stored information on ingredients and dishes, a means for recognizing emotions from the user's facial expressions, tone of voice, and text data, and a means for adjusting suggestions based on the recognized emotions. This allows for more refined diet management for the user and makes suggestions tailored to their emotions. Furthermore, it enables automated repurchase suggestions and online ordering, improving user convenience.

[1620] "Means for receiving images of meals taken by the user" is a function for collecting image data of meals taken using a device such as a smartphone or digital camera and sending it to a server or cloud.

[1621] The "means for analyzing the image and identifying information about ingredients and dishes" is a function that uses machine learning algorithms and image recognition technology to identify the types of ingredients and dishes contained in the image based on the received image data.

[1622] The "means for saving information on the identified ingredients and dishes in a database" is a function for recording and accumulating the analyzed ingredients and dish types and related nutritional information in a database.

[1623] "Means for managing the user's diet based on stored information on ingredients and dishes" refers to a function that uses information stored in the database to manage the user's dietary history, nutritional intake, calories, etc., and to suggest appropriate meals and manage health.

[1624] "Means for recognizing emotions from a user's facial expression, tone of voice, and text data" is a function for analyzing emotions from a user's facial expression data, tone of voice, and text content, and identifying that state.

[1625] The "means for adjusting the content of suggestions based on the recognized emotion" is a function for using the recognized emotion information to appropriately change the content of suggestions for meals and ingredients to the user, and providing services that correspond to the user's emotional state.

[1626] The "means for suggesting ingredients and seasonings that need to be repurchased" is a function that identifies ingredients and seasonings that are in short supply based on the user's meal record and information on ingredients consumed, and suggests to the user that they be repurchased.

[1627] The "means for placing an online order based on the suggestions" is a function that provides a link or interface for the user to place an order for the suggested ingredients or seasonings at an online shop.

[1628] The "means for providing links related to delivery services" is a function that provides users with links for ordering the suggested dishes and ingredients through delivery services.

[1629] In one embodiment of the present invention, a system is used that receives images of food taken by a user, identifies ingredients and dishes, and makes suggestions based on the user's emotional state. This system is realized using the user's smartphone, a server, a database, an emotion recognition engine, etc.

[1630] 1. Meal photo recognition and management

[1631] User

[1632] Users use a smartphone app to take a photo of their meal and tap the "send" button.

[1633] Terminal

[1634] The device compresses the captured image and sends it to the server, where the image data is sent in the appropriate format (e.g., JPEG or PNG).

[1635] server

[1636] The server inputs the received image data into an image recognition module and uses machine learning algorithms (e.g., CNN) to identify the ingredients and dishes in the image. The identified ingredients and dishes are stored in a database. This information includes the names of the ingredients, the names of the dishes, the serving sizes, calories, and other nutritional information. The stored information is also used to update the user's food log and manage calorie and macronutrient intake.

[1637] Specific examples

[1638] For example, a user takes a photo of their lunch of "teriyaki chicken," "rice," and "salad," and the device sends the photo to the server. The server analyzes the image, identifies the chicken, rice, lettuce, and tomato, and stores their nutritional information in the user's database. This data is used to manage the user's nutritional intake.

[1639] 2. Suggestions for repurchasing ingredients and seasonings

[1640] server

[1641] The server periodically analyzes the user's food log and checks the food ingredient data consumed. This analysis includes the amount consumed over a specific period (e.g., one week). If an ingredient or seasoning exceeds a threshold, it determines that it needs to be repurchased. It generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message. The message includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1642] Terminal

[1643] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[1644] User

[1645] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1646] Specific examples

[1647] The server analyzes the recent food log and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[1648] 3. Recipe suggestions

[1649] server

[1650] The server analyzes the user's food records and preference data, including the user's favorite ingredients and nutritional balance data. It automatically generates recipes that take nutritional balance into consideration and selects recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences. It then creates a message to notify the user of the generated recipe. The message contains detailed information such as the recipe name, required ingredients, quantities, and cooking instructions.

[1651] Terminal

[1652] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1653] User

[1654] The user confirms the notification and views the details of the suggested recipe. The user then prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1655] Specific examples

[1656] The server analyzes the user's recent food log and detects that they are iron deficient. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user checks the notification and cooks the dish according to the recipe.

[1657] 4. Incorporating an Emotional Engine

[1658] server

[1659] The server uses an emotion engine that recognizes the user's emotions from their facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state and adjust the recommendations based on the recognized emotion. For example, if the user is feeling stressed, the server will suggest ingredients and dishes that have a relaxing effect.

[1660] Terminal

[1661] The terminal receives the emotion-based suggestion message from the server and notifies the user.

[1662] User

[1663] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[1664] Specific examples

[1665] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[1666] Prompt Sentence Examples

[1667] "Analyze the user's food photos and identify the food and ingredients. Also, determine the user's emotions from their facial expressions and tone of voice, and suggest ingredients and recipes based on that."

[1668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1669] Step 1:

[1670] A user takes a photo of their meal using their smartphone and taps the "send" button.

[1671] Input: Image file of the food you have taken

[1672] Output: Food image data sent to the server

[1673] Specific operation: Take a photo of your meal using the smartphone's camera app, and send the image to the server using the "Send" button in the application.

[1674] Step 2:

[1675] The device compresses the images it takes and sends them to the server.

[1676] Input: Image file of the photographed meal

[1677] Output: Compressed food image data (JPEG or PNG format)

[1678] Specific operation: The device compresses the image file into the appropriate format and sends it to the specified server address.

[1679] Step 3:

[1680] The image data received by the server is input into an image recognition module to identify the ingredients and the dish.

[1681] Input: Compressed food image data

[1682] Output: Identified ingredients and dish information (e.g., ingredient name, dish name)

[1683] Specific operation: The received image data is input into a machine learning algorithm (e.g., a CNN model) to identify the ingredients and dishes in the image.

[1684] Step 4:

[1685] The server stores the identified ingredients and dish information in a database.

[1686] Input: Identified ingredients and dish information

[1687] Output: Updated food record in the database

[1688] Specific behavior: Writes the identified ingredients and dish data to a database and updates the user's meal record.

[1689] Step 5:

[1690] The server recognizes the user's emotions from their facial expressions, tone of voice, and text data.

[1691] Input: User's facial expression data, voice data, text data

[1692] Output: Perceived emotional state (e.g., stress, happiness)

[1693] Specific operation: Facial expressions and tone of voice collected by the smartphone's camera and microphone are input into an emotion recognition engine to identify the user's emotional state.

[1694] Step 6:

[1695] The server adjusts the suggestions based on the perceived sentiment.

[1696] Input: Perceived emotional state

[1697] Output: Tailored recommendations (e.g., ingredients and dishes with relaxing effects)

[1698] What it does: It applies algorithms that change the food and recipe suggestions provided to users based on their emotional state.

[1699] Step 7:

[1700] The server notifies the user of suggested recipes and repurchase suggestions.

[1701] Input: Adjusted recommendations, ingredients and seasonings that need to be repurchased

[1702] Output: A notification message displayed on the user's terminal.

[1703] Specific behavior: A suggestion message is sent to the user's smartphone as a push notification, and the user receives the notification.

[1704] Step 8:

[1705] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions.

[1706] Input: Notification content (recipe information, repurchase link)

[1707] Output: Order completed at online store

[1708] Specific behavior: The user taps the notification to be taken to the linked online shop, where they can add the suggested ingredients and seasonings to their cart and place an order.

[1709] Step 9:

[1710] The server generates a message suggesting ingredients and seasonings that need to be repurchased and sends it to the user.

[1711] Input: Food log and food consumption data

[1712] Output: Suggestion message (list of missing ingredients and seasonings)

[1713] Specific operation: Analyzes food records, identifies ingredients and seasonings consumed frequently, and generates suggestion messages based on that information and sends them to the user.

[1714] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1715] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1716] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1717] [Fourth embodiment]

[1718] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1719] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1720] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1721] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1722] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1723] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1724] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1725] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1726] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1729] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1730] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1731] The present invention relates to a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. An embodiment of this system will be described in detail below.

[1732] Meal photo recognition and management

[1733] User

[1734] The user takes a photo of the meal and launches the application, which saves the image of the meal on the user's device.

[1735] Terminal

[1736] The device sends the images taken by the user to the server, where the image data is compressed in an appropriate format to ensure efficient data transfer.

[1737] server

[1738] The server analyzes the images received from the device using machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify ingredients and dishes in the images.

[1739] The identified ingredients and dish information is stored in a database, which includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1740] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[1741] Specific examples

[1742] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[1743] Repurchase suggestions for ingredients and seasonings

[1744] server

[1745] The server periodically analyzes the user's food log and consumed food ingredient data, for example, to identify ingredients or condiments consumed in amounts above a certain threshold.

[1746] The server lists ingredients and seasonings that need to be repurchased and generates a suggestion message to notify the user at an appropriate time.

[1747] Terminal

[1748] The device then sends a repurchase suggestion message to the user, which includes the suggested ingredients and seasonings and a link to an online shop for repurchasing.

[1749] User

[1750] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[1751] Specific examples

[1752] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[1753] Recipe suggestions

[1754] server

[1755] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, including the user's preferred ingredients and past meal history.

[1756] The server creates a message to notify the user of the created recipe, which includes detailed information about the ingredients, quantities, cooking steps, etc.

[1757] Terminal

[1758] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[1759] User

[1760] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to follow a nutritionally balanced diet.

[1761] Specific examples

[1762] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[1763] conclusion

[1764] In this way, the system analyzes photos of meals taken by the user to manage their diet, suggests repurchases of consumed ingredients and seasonings, and provides nutritionally balanced recipes. This allows users to efficiently manage their health while reducing the effort required for managing and purchasing ingredients.

[1765] The processing flow will be explained below.

[1766] Meal photo recognition and management

[1767] Step 1:

[1768] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[1769] Step 2:

[1770] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[1771] Step 3:

[1772] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[1773] Step 4:

[1774] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1775] Step 5:

[1776] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[1777] Repurchase suggestions for ingredients and seasonings

[1778] Step 1:

[1779] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[1780] Step 2:

[1781] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[1782] Step 3:

[1783] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[1784] Step 4:

[1785] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the ingredients and seasonings needed and a link to repurchase them.

[1786] Step 5:

[1787] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[1788] Recipe suggestions

[1789] Step 1:

[1790] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[1791] Step 2:

[1792] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[1793] Step 3:

[1794] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[1795] Step 4:

[1796] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[1797] Step 5:

[1798] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[1799] summary

[1800] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes, allowing them to efficiently manage their daily health.

[1801] Example 1

[1802] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1803] When users manage their daily diet, it is time-consuming to understand the details of ingredients and dishes and update records based on accurate nutritional information. Furthermore, knowing when to repurchase consumed ingredients and seasonings and suggesting recipes that take nutritional balance into consideration are also significant burdens for individual users. This makes ongoing health management difficult, creating a challenge.

[1804] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1805] In this invention, the server includes means for analyzing images of meals taken by the user to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for updating the user's meal record based on the stored information, means for analyzing the user's meal record and consumed ingredients and suggesting ingredients and seasonings that need to be repurchased, and means for suggesting recipes that take the user's nutritional balance into consideration based on the user's meal record and preference data. This allows the user to continuously and easily manage their diet, appropriately repurchase consumed ingredients and seasonings, and prepare meals that take nutritional balance into consideration.

[1806] A "user" is someone who uses the system to access functions such as dietary management, repurchasing ingredients, and recipe suggestions.

[1807] A "terminal" refers to an electronic device such as a smartphone, tablet, or PC used by a user to take photos of meals, send images, receive notifications, and so on.

[1808] A "server" is a computer system that analyzes image data sent from a terminal, stores the results in a database, and makes various suggestions to the user.

[1809] "Image data" refers to digital information of a photograph of a meal taken by a user, which is analyzed by the server.

[1810] "Machine learning algorithms" refer to artificial intelligence techniques used by the server to analyze image data, including convolutional neural networks (CNNs).

[1811] "Ingredients" are the individual food components in a meal that contain nutritional information.

[1812] A "dish" is a meal prepared according to a specific recipe and consisting of a combination of ingredients.

[1813] A "database" is an electronic recording device that stores information on ingredients and dishes, user meal records, preference data, and so on.

[1814] A "repurchase suggestion message" is a notification message sent by the server after analyzing the user's consumption data regarding ingredients or seasonings that need to be repurchased.

[1815] "Nutritional balance" refers to the distribution of each nutrient in the user's diet, and indicates whether it is balanced for health management.

[1816] "Recipe suggestions" refers to the server providing users with cooking methods that take nutritional balance into consideration based on the user's meal records and preference data.

[1817] "Notification" refers to the means by which a device communicates information such as repurchase suggestions or recipe suggestions to a user, and includes push notifications.

[1818] This invention relates to a system that analyzes images of meals taken by users, identifies ingredients and dishes, and performs dietary management, suggests repurchasing ingredients, and suggests recipes based on nutritional balance. The system of the present invention is configured to achieve the following three main functions:

[1819] 1. Meal photo recognition and management

[1820] User

[1821] To record the contents of their meals, users take photos of their meals using the device's camera. The images are then saved directly to the device.

[1822] Terminal

[1823] The device has the ability to compress the image data captured by the user into an appropriate format (e.g., JPEG or PNG) and send it to the server. Specifically, it uses a compression algorithm to ensure efficient data transfer.

[1824] server

[1825] The server receives the image data sent from the device and analyzes it using a machine learning algorithm (e.g., convolutional neural network: CNN) to identify the ingredients and dishes.

[1826] The server stores the identified ingredients and dish information (e.g., ingredient name, dish name, portion size, calories, nutritional information) in a database.

[1827] Based on the saved information, it has the function of updating the user's food record and calculating and saving the total calories and nutritional components ingested.

[1828] Specific examples

[1829] A user takes a photo of their lunch: teriyaki chicken, rice, and salad. The device compresses the photo into JPEG format and sends it to the server. The server performs image analysis, identifies the ingredients (chicken, rice, lettuce, and tomato), and stores their nutritional information in a database. Finally, the user's food record is updated, enabling accurate nutritional management.

[1830] 2. Suggestions for repurchasing ingredients and seasonings

[1831] server

[1832] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[1833] The server generates a repurchase suggestion message regarding ingredients and seasonings that need to be repurchased and transmits it to the user's terminal.

[1834] Terminal

[1835] The device displays the repurchase suggestion message sent from the server to the user as a push notification, which includes the name of the suggested ingredient or seasoning and a link to the online shop for repurchase.

[1836] User

[1837] The user can check the notification and order ingredients and seasonings from the online shop as needed, thereby preventing shortages of ingredients and seasonings.

[1838] Specific examples

[1839] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps a link to order soy sauce from an online shop.

[1840] 3. Recipe suggestions

[1841] server

[1842] The server analyzes the user's food records and preference data to generate recipes that take into account the user's nutritional balance, and suggests appropriate recipes based on the user's preferred ingredients and past meal history.

[1843] Create a message to notify the user of the generated recipe, which includes detailed information such as ingredients, quantities, and cooking steps.

[1844] Terminal

[1845] The device notifies the user of the recipe suggestions received from the server, and the user can open the notification to check the details of the suggested recipe.

[1846] User

[1847] Users can refer to the suggested recipes, prepare the necessary ingredients, and cook, making it easy to eat a nutritionally balanced diet.

[1848] Specific examples

[1849] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their iron.

[1850] conclusion

[1851] This system analyzes images of meals taken by users and provides dietary management, suggestions for repurchasing ingredients, and recipe suggestions based on nutritional balance, allowing users to efficiently manage their health and reducing the effort required for managing and purchasing ingredients, making ongoing health management easier.

[1852] Prompt Sentence Examples

[1853] "Please explain in natural language the program for a system that takes a photo of a meal using an app, sends it to a server, identifies ingredients and dishes from the analysis results, and updates the food record based on the nutritional information. Please include specific examples."

[1854] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1855] Step 1:

[1856] The user takes a photo of their meal, and the image is saved in the device's internal storage.

[1857] Input: A user-taken image of a meal.

[1858] How it works: You take a photo using the camera app on your smartphone or tablet, which creates an image file.

[1859] Output: Image file saved in the device's internal storage.

[1860] Step 2:

[1861] The device compresses the image data and sends it to the server. The device uses a compression format such as JPEG or PNG.

[1862] Input: An image file stored in the device's internal storage.

[1863] How it works: An image compression algorithm is run on the device to generate a compressed image file, which is then sent over the internet to a server.

[1864] Output: Compressed image data sent to the server.

[1865] Step 3:

[1866] The server analyzes the received image data and uses machine learning algorithms (e.g., convolutional neural networks (CNNs)) to identify the ingredients and dishes in the image.

[1867] Input: Compressed image data sent from the terminal.

[1868] How it works: Machine learning algorithms run on the server to analyze the image, identify ingredients and dishes, and extract information about them.

[1869] Output: Information on identified ingredients and dishes (ingredients, dish name, portion size, calories, nutritional information, etc.).

[1870] Step 4:

[1871] The server stores the identified ingredients and dish information in a database.

[1872] Input: Ingredient and dish information identified by the analysis.

[1873] Operation: The analysis results are written and saved in the server's database system.

[1874] Output: Ingredient and dish information stored in a database.

[1875] Step 5:

[1876] The server updates the user's food record based on the stored information, calculates the total calories and nutritional information, and updates the database.

[1877] Input: Identified ingredients and dish information.

[1878] How it works: The server runs a nutrition calculation algorithm that adds calorie and nutrient totals to each user's food log.

[1879] Output: Updated user's meal log.

[1880] Step 6:

[1881] The server periodically analyzes the user's meal record and data on consumed ingredients to identify ingredients and seasonings that need to be repurchased.

[1882] Input: User's food log and consumed ingredients data.

[1883] How it works: The server runs a data analysis algorithm to identify ingredients and seasonings that are being consumed most frequently.

[1884] Output: A list of ingredients and seasonings that need to be repurchased.

[1885] Step 7:

[1886] The server generates a repurchase offer message and sends it to the user's terminal.

[1887] Input: A list of ingredients and seasonings that need to be repurchased.

[1888] Behavior: The server generates a proposal message and sends it to the device as a push notification.

[1889] Output: Repurchase offer message sent to the user's device.

[1890] Step 8:

[1891] The user checks the repurchase suggestion message and orders ingredients and seasonings from the online shop as needed.

[1892] Input: Repurchase suggestion message.

[1893] Action: The user sees the notification and taps the link to go to the online store and order the ingredients and seasonings they need.

[1894] Output: The ingredients and seasonings for the completed order.

[1895] Step 9:

[1896] The server analyzes the user's food records and preference data and generates recipes that take nutritional balance into consideration.

[1897] Input: User's food log and preference data.

[1898] How it works: Data analysis is performed on the server to generate nutritionally balanced recipes that are optimal for the user.

[1899] Output: Nutritionally balanced recipes.

[1900] Step 10:

[1901] The server notifies the user of the generated recipe.

[1902] Input: The generated recipe.

[1903] Operation: The server generates a recipe notification message and sends it to the device as a push notification.

[1904] Output: Recipe notification sent to the user's device.

[1905] Step 11:

[1906] The user checks the recipe notification and cooks using the suggested recipe as a reference.

[1907] Enter: recipe notification.

[1908] What happens: The user opens the notification, prepares the ingredients, and follows the recipe displayed to cook the food.

[1909] Output: A nutritionally balanced diet.

[1910] The above is the flow of processing of the program of this system.

[1911] (Application example 1)

[1912] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1913] In modern society, it is important for users to manage their health on a daily basis, but it is difficult to efficiently manage their diet and adjust nutritional balance in a busy lifestyle. There is also a lack of comprehensive systems that support overall dietary habits, such as managing food purchases and suggesting new recipes. Therefore, there is a need for a system that can identify information about ingredients and dishes based on photos of meals taken by users, and then use that information to manage their diet, suggest repurchasing ingredients, suggest recipes that take nutritional balance into consideration, and even deliver health-related content.

[1914] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1915] In this invention, the server includes a means for receiving images of meals taken by a user, a means for analyzing the images to identify ingredients and dish information, a means for storing the identified ingredient and dish information in a database, a means for managing the user's diet based on the stored ingredient and dish information, and a means for delivering health-related content. This allows users to automate their daily diet management and live a nutritionally balanced and healthy lifestyle. Furthermore, the server includes a means for suggesting ingredients and seasonings that need to be repurchased, a means for suggesting recipes based on the user's preference data, and a means for suggesting personalized health content using a generative AI model, enabling efficient comprehensive health management.

[1916] "User" refers to any individual or organization that uses this system.

[1917] "Meal image" refers to a digital image taken by a user that visually records the contents of a meal.

[1918] "Analysis" refers to the process of identifying ingredients and dish information from received food images, primarily using machine learning algorithms.

[1919] "Ingredient and dish information" is data indicating the components of a meal identified through image analysis, and includes the names of ingredients, dish names, quantities, nutritional components, etc.

[1920] "Database" refers to a digital storage system that organizes and stores information about identified ingredients and dishes so that it can be retrieved and updated later.

[1921] "Diet management" refers to the activity of tracking a user's diet and monitoring and adjusting the balance of calories and nutrients ingested.

[1922] "Health-related content" refers to digital content such as videos and articles that provide customized information and advice based on a user's health and diet.

[1923] "Repurchase suggestions" refers to notifications or messages that identify ingredients or seasonings that are missing based on the user's meal records and suggest purchasing them.

[1924] "Online ordering" refers to the process by which a user purchases suggested ingredients and seasonings through an online shopping system.

[1925] "Preference data" refers to personal data about ingredients or dishes selected by the user in the past, or the user's preferences.

[1926] "Recipe suggestion" refers to the activity of informing users of nutritionally balanced recipes based on their food records and preference data.

[1927] "Generative AI model" refers to the artificial intelligence algorithm used to analyze a user's eating habits and suggest personalized health content.

[1928] "Personalized health content" refers to health-related information and advice that is customized for a particular user based on the user's individual food records and nutritional status.

[1929] This invention provides a system for managing meals by analyzing images of meals taken by users and identifying information about ingredients and dishes. This system aims to provide comprehensive health management, including suggestions for repurchasing ingredients, recipe suggestions that take nutritional balance into consideration, and the distribution of health-related content.

[1930] 1. System Overview

[1931] The system consists of three main parts: user terminals, servers, and databases.

[1932] 2. User Device

[1933] The user device is a smartphone or tablet with a camera. The user takes a photo of the meal and launches a dedicated application. This application has the function of acquiring the captured image, compressing it in an appropriate format, and sending it to the server.

[1934] 3. Server

[1935] The server has the following functions:

[1936] Image analysis function

[1937] The server analyzes the received meal images using a convolutional neural network (CNN) to identify the ingredients and dishes in the image. The analysis results include the names of the ingredients, the dish, the serving size, calories, and other nutritional information.

[1938] Database Management Functions

[1939] The identified ingredients and dish information are stored in a database that holds the user's meal record, preference data, and data on ingredients consumed.

[1940] Meal management function

[1941] The server uses the stored information to update the user's food record, which includes totals for calories, protein, carbohydrates, fat, and other nutrients.

[1942] Repurchase suggestion function

[1943] The server analyzes the user's meal log and identifies ingredients and seasonings that need to be repurchased. Once identified, the server notifies the user and provides a link to encourage online ordering.

[1944] Recipe suggestion function

[1945] The server analyzes the user's food records and preference data, and generates recipes that take into account the user's nutritional balance. The generated recipes are then notified to the user.

[1946] Health-related content distribution function

[1947] Using a generative AI model, it analyzes users' eating habits and suggests personalized health-related content, including videos and articles related to staying healthy.

[1948] 4. Specific Examples

[1949] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to a server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is stored in the user's database. The server updates the user's food record based on the nutritional information of these ingredients, providing appropriate nutritional management. The server also identifies ingredients that are in short supply and generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to repurchase?" The recipe suggestion function then sends the user a notification saying, "You are low on iron. We suggest a recipe using spinach and beef." The generative AI model also analyzes the user's eating habits and delivers "advice videos for increasing specific nutrients."

[1950] 5. Examples of Prompt Statements

[1951] "Based on photos of meals taken by users with their smartphones, please use a CNN model to analyze what ingredients and dishes are included, and upload the results to a database on the cloud. Also, please make sure that dietary management and recipe suggestions are performed based on the user's nutritional balance."

[1952] This system enables individual users to lead a nutritionally balanced and healthy life while automating their daily dietary management, and also enables efficient comprehensive health management.

[1953] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1954] Step 1:

[1955] The user takes a photo of the meal.

[1956] Input: Food image

[1957] Output: Raw image files saved on the device

[1958] Specific operation: The user takes a photo of the meal using the camera app on their smartphone and launches a dedicated application.

[1959] Step 2:

[1960] The device sends the captured image to the server.

[1961] Input: Image file saved on the device

[1962] Output: Compressed image data sent to the server

[1963] Specific operation: The application compresses the image file in the appropriate format and sends it to the server using a data transfer protocol (e.g., HTTP).

[1964] Step 3:

[1965] The server analyzes the received image.

[1966] Input: Compressed image data sent

[1967] Output: Analysis results (information on identified ingredients and dishes)

[1968] How it works: The server uses a convolutional neural network (CNN) to analyze the image and identify ingredients and dishes. The analysis results include the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[1969] Step 4:

[1970] The server stores the identified ingredients and dish information in a database.

[1971] Input: Analysis results (identified ingredients and dish information)

[1972] Output: Ingredient and dish information stored in a database

[1973] Specific operation: The server executes queries to store the analysis results in an SQL or NoSQL database.

[1974] Step 5:

[1975] The server manages dietary management based on the stored information.

[1976] Input: Ingredient and dish information stored in a database

[1977] Output: Data required for dietary management, updated user diet record

[1978] Specific operation: The server retrieves the stored information, calculates the total calories and nutritional intake, and updates the user's food record.

[1979] Step 6:

[1980] The server analyzes the user's meal log and the ingredients consumed.

[1981] Input: User's food records and consumed ingredients data stored in a database

[1982] Output: List of ingredients and seasonings that need to be repurchased

[1983] Specific operation: The server uses an algorithm to analyze the user's food records and identify ingredients and seasonings whose consumption exceeds a certain threshold.

[1984] Step 7:

[1985] The server generates a repurchase offer message and sends it to the terminal.

[1986] Input: List of ingredients and seasonings that need to be repurchased

[1987] Output: Repurchase suggestion message

[1988] Specific operation: The server generates a proposal message and sends it to the user's device via push notification or email.

[1989] Step 8:

[1990] The server analyzes the user's food records and preference data.

[1991] Input: Ingredient and dish information stored in the database, user preference data

[1992] Output: Generate recipes that take nutritional balance into consideration

[1993] Specific operation: The server generates nutritionally balanced recipes based on the user's preference data and food records, and uses a generative AI model to suggest personalized health content.

[1994] Step 9:

[1995] The device will notify you of suggested recipes and health-related content.

[1996] Input: Generated recipes and personalized health content

[1997] Output: Notifications and application screen display

[1998] Specific behavior: The device displays the received notification and allows the user to view recipes and health-related content.

[1999] These processing steps enable users to efficiently manage their diet and lead a nutritionally balanced and healthy life. Furthermore, the generative AI model provides personalized health content, allowing users to receive advice based on their individual health conditions.

[2000] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2001] The present invention is a system that analyzes images of meals taken by a user to identify ingredients and dishes, and then performs dietary management, suggests repurchasing ingredients, and suggests recipes that take nutritional balance into consideration based on this information. By further combining this with an emotion engine, the system makes suggestions based on the user's emotions. An embodiment of this system will be described in detail below.

[2002] Meal photo recognition and management

[2003] User

[2004] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[2005] Terminal

[2006] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[2007] server

[2008] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[2009] The identified ingredients and dish information is stored in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[2010] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[2011] Specific examples

[2012] A user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. The device sends the photo to the server, which analyzes the image. As a result of the analysis, chicken, rice, lettuce, and tomato are identified, and their nutritional information is saved in the user's database. Based on the nutritional information of these ingredients, the server updates the user's food record and provides appropriate nutritional management.

[2013] Repurchase suggestions for ingredients and seasonings

[2014] server

[2015] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[2016] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[2017] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[2018] Terminal

[2019] The device will then send a repurchase suggestion message to the user, displaying the suggested ingredients and seasonings and a link to repurchase.

[2020] User

[2021] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[2022] Specific examples

[2023] The server analyzes recent food records and detects that soy sauce consumption is increasing. The server generates a repurchase suggestion message saying, "Your soy sauce stock is low. Would you like to purchase more?" and sends it to the device. The user checks the notification and taps the link to order soy sauce from the online shop.

[2024] Recipe suggestions

[2025] server

[2026] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[2027] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[2028] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[2029] Terminal

[2030] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[2031] User

[2032] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[2033] Specific examples

[2034] The server analyzes the user's recent food records and detects that they are low in iron. The server generates a nutritionally balanced recipe using spinach and beef and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to ensure adequate iron replenishment.

[2035] Incorporating an emotion engine

[2036] server

[2037] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[2038] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[2039] Terminal

[2040] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[2041] User

[2042] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[2043] Specific examples

[2044] When a user is tired from work, the app suggests a recipe for "Teriyaki Chicken and Vegetable Soup." The emotion engine recognizes that the user is tired and also suggests a relaxing herbal tea. The user can check the notification and make the herbal tea to relax.

[2045] conclusion

[2046] This invention provides a unique function that combines an emotion engine with a system that analyzes images of meals taken by users to manage their diet, suggest repurchasing ingredients, and suggest recipes based on nutritional balance. This enables health management according to the user's emotional state and makes more personalized suggestions.

[2047] The processing flow will be explained below.

[2048] Meal photo recognition and management

[2049] Step 1:

[2050] A user takes a photo of their meal using a smartphone app. After taking the photo, the user taps the "Send Photo" button.

[2051] Step 2:

[2052] The device takes a picture, compresses it, and sends it to the server, where it sends the image data in the appropriate format (e.g., JPEG or PNG).

[2053] Step 3:

[2054] The server inputs the received image data into an image recognition module, which uses machine learning algorithms (e.g., CNN) to identify ingredients and dishes in the image.

[2055] Step 4:

[2056] The server stores the identified ingredients and dish information in a database, including the names of ingredients, dishes, serving sizes, calories, and other nutritional information.

[2057] Step 5:

[2058] The server uses the stored information to update the user's food log, which includes total calories, protein, carbohydrates, and fat intake.

[2059] Repurchase suggestions for ingredients and seasonings

[2060] Step 1:

[2061] The server periodically analyzes the user's food log and checks the food consumption data, including the amount consumed over a specific period (e.g., a week).

[2062] Step 2:

[2063] The server identifies ingredients and seasonings that have exceeded a consumption threshold, and determines that the identified ingredients and seasonings need to be repurchased.

[2064] Step 3:

[2065] The server generates a list of ingredients and seasonings that need to be repurchased and creates a suggestion message, which includes the specific names of the ingredients and seasonings and a link to repurchase them.

[2066] Step 4:

[2067] The device receives a repurchase suggestion message from the server and notifies the user, displaying information about the suggested ingredients and seasonings and a link to repurchase them.

[2068] Step 5:

[2069] The user checks the notification and orders ingredients and seasonings from the online shop based on the suggestions. The user clicks the link to go to the online shop's order page.

[2070] Recipe suggestions

[2071] Step 1:

[2072] The server analyzes the user's food records and preference data, including the user's favorite foods and nutritional balance.

[2073] Step 2:

[2074] The server automatically generates nutritionally balanced recipes, selecting recipes that reflect the user's nutritional deficiency data (e.g., iron deficiency) and preferences.

[2075] Step 3:

[2076] The server creates a message to notify the user of the recipe, including the recipe name, ingredients, quantities, cooking instructions, etc.

[2077] Step 4:

[2078] The device receives the recipe suggestion message from the server and notifies the user. The notification is displayed on the user's device in the form of a push notification.

[2079] Step 5:

[2080] The user confirms the notification and views the details of the suggested recipe. The user prepares the ingredients according to the recipe and follows the cooking instructions to cook the dish.

[2081] Incorporating an emotion engine

[2082] Step 1:

[2083] The server uses an emotion engine that recognizes emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state.

[2084] Step 2:

[2085] The server adjusts the suggestions based on the emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest ingredients and dishes that have a relaxing effect.

[2086] Step 3:

[2087] The server creates emotion-based suggestion messages, which include suggestions for ingredients and dishes based on the user's emotional state.

[2088] Step 4:

[2089] The terminal receives the suggestion message based on the emotion from the server and notifies the user.

[2090] Step 5:

[2091] Users can view emotion-based suggestions and act on them, for example, cooking a meal that will relieve stress.

[2092] Specific examples

[2093] Example 1: Food photo recognition

[2094] A user takes a photo of "toast and coffee" for breakfast. The device sends the photo to the server, which analyzes the photo and identifies the toast and coffee. This nutritional information is stored in a database and the user's food record is updated.

[2095] Example 2: Repurchase offer

[2096] The server analyzes the user's food log and detects that milk consumption is increasing. The server generates a repurchase suggestion message saying "Milk stock is low. Would you like to repurchase?" and sends it to the device. The user checks the notification and taps the link to order milk from the online shop.

[2097] Example 3: Recipe suggestions

[2098] The server analyzes the user's food records and detects a vitamin D deficiency. The server generates a nutritionally balanced recipe using fish and spinach and sends a notification with specific cooking instructions to the device. The user can check the notification and cook the dish according to the recipe to properly replenish their vitamin D.

[2099] Example 4: Emotion engine suggestions

[2100] While a user is using a smartphone app, the emotion engine recognizes that the user is tired. The server generates a suggestion message saying, "How about some relaxing herbal tea?" and sends it to the device. The user can check the notification and prepare the suggested herbal tea to relax.

[2101] conclusion

[2102] By following the above steps, users can easily record photos of their meals, receive recommendations for repurchasing ingredients and seasonings, and receive nutritionally balanced recipes. Furthermore, by incorporating an emotion engine, suggestions can be made based on the user's emotional state, enabling even more personalized support.

[2103] Example 2

[2104] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2105] Conventional dietary management systems analyze nutritional information and suggest repurchasing ingredients, but are unable to provide suggestions that take the user's emotional state into account. As a result, personalized suggestions based on the user's mental state and preferences are insufficient, limiting the effectiveness of health management. The present invention aims to realize more personalized dietary and health management by recognizing the user's emotions in real time and adjusting the suggestions based on this.

[2106] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2107] In this invention, the server includes means for receiving images of meals taken by a user, means for analyzing the images to identify information about ingredients and dishes, means for storing the identified information about ingredients and dishes in a database, means for managing the user's meals based on the stored information about ingredients and dishes, means for recognizing the user's emotions from facial expressions, tone of voice, and text data, and means for adjusting the content of suggestions based on the recognized emotions, thereby enabling personalized suggestions based on the user's emotional state.

[2108] A "user" is an individual who uses the system to take photos of their meals and use those photos for analysis.

[2109] The "server" is a central processing unit that receives image data sent by users and performs analysis and proposals.

[2110] A "terminal" is a device used by a user, such as a smartphone or tablet, that has the function of sending captured images to a server.

[2111] A "meal image" is a photograph of a meal taken by a user, and serves as data for identifying ingredients and dishes.

[2112] "Analysis" refers to the process performed to identify ingredients and dish information from the received food image.

[2113] "Ingredient and dish information" refers to the names of ingredients, dishes, amounts, calories, and other nutritional information identified through analysis.

[2114] "Database" refers to an information management system for storing information on identified ingredients and dishes.

[2115] "Dietary management" refers to the act of managing the calorie and nutrient intake of a user based on stored information about ingredients and dishes.

[2116] "Means for recognizing emotions" refers to technology that has the ability to analyze a user's emotions from their facial expressions, tone of voice, and text data.

[2117] "Means to adjust suggestion content" refers to a function that appropriately changes the content of meal management, repurchase, and recipe suggestions based on recognized emotions.

[2118] The invention begins with a user taking a picture of a meal using a device such as a smartphone or tablet and sending the image to a server. The device compresses the image using a library such as Python's Pillow and sends it to the server in an appropriate format (e.g., JPEG or PNG). The server then inputs the received image data into an image recognition module, which uses a convolutional neural network (CNN) and an AI model.

[2119] The image recognition module uses machine learning algorithms to identify ingredient and dish information from the submitted image. The identified information includes the names of ingredients, dishes, serving sizes, calories, and other nutritional information. This information is stored in a database on the server. This eliminates the need for users to record their meals themselves, allowing for automated diet management.

[2120] The present invention also incorporates an emotion engine, allowing the server to recognize emotions from the user's facial expressions, tone of voice, and text data. The emotion engine can analyze the user's real-time emotional state, and if the user is feeling stressed or tired, for example, it will suggest ingredients and dishes that have a relaxing effect.

[2121] For example, a user takes a photo of "teriyaki chicken," "rice," and "salad" for lunch. When the device sends the photo to the server, the server's image recognition module analyzes it and identifies the ingredients, such as chicken, rice, lettuce, and tomato. The nutritional information for these ingredients is then stored in a database, and the user's meal record is updated. Furthermore, the emotion engine analyzes the user's emotional state and, if necessary, suggests relaxing herbal teas.

[2122] Examples of prompt sentences include the following:

[2123] Ingredient identification model

[2124] "Identify the ingredients and name of the dish based on a photo of the meal taken by the user."

[2125] Emotion Engine

[2126] "Analyze the user's emotions from facial expressions, tone of voice, and text data, and generate appropriate suggestions."

[2127] In this way, by appropriately utilizing the generative AI model and prompt sentences, personalized suggestions based on the user's condition become possible. This system enables efficient and effective dietary management for users, and also realizes health management that takes into account the user's emotional state.

[2128] The flow of the identification process in the second embodi...

Claims

1. means for receiving an image of a meal taken by a user; means for analyzing the image to identify information about ingredients and dishes; a means for storing information on the identified ingredients and dishes in a database; A means for managing the user's diet based on the stored information on ingredients and dishes; A system including:

2. A means for analyzing a user's meal record and consumed ingredients and suggesting ingredients and seasonings that need to be repurchased; means for placing an online order based on said suggestions; The system of claim 1 further comprising:

3. A means for suggesting recipes that take into consideration the nutritional balance of the user based on the user's food records and preference data; means for notifying a user of the suggested recipe; The system of claim 1 further comprising:

Images