system

Abstract

We provide the system. [Solution] A means for receiving food images taken by a user, analyzing the images, and identifying food component data, Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data, We select a service that proposes nutritionally balanced menus to supplement identified nutrient deficiencies, and provide this service to the user. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In daily life, it is difficult for many people to ingest necessary nutrients in a balanced manner. This difficulty is due to the fact that it is not easy to analyze the diet content and accurately judge the lacking nutrients. Especially for people with an unbalanced diet or those who desire a balanced diet but do not know how, it is difficult to maintain a healthy diet.

Means for Solving the Problems

[0005] This invention provides a system that automatically analyzes nutrients from images of food taken by a user, identifies deficient nutrients based on the user's individual health data, and provides optimal meal suggestions to supplement those deficiencies. Specifically, it uses a deep learning algorithm to accurately identify food components, and considers the user's age, gender, and past health history to generate meal suggestions to supplement deficient nutrients. This system enables users to effectively consume nutritionally balanced meals.

[0006] "User-submitted food images" refer to images of the food a user has taken using their device and obtained as image data.

[0007] "Methods for identifying food component data by analyzing images" refers to methods that use image analysis technology to recognize the type of food and the nutritional components it contains, and then identify data based on that.

[0008] "A method for identifying deficient nutrients based on analyzed food component data and individual user health data" refers to a method that combines acquired component data with user-specific health information for analysis to identify which nutrients are needed.

[0009] "A means of generating and presenting dietary suggestions to users to supplement identified nutrient deficiencies" refers to a method of creating menus or recipes using foods that correspond to the identified nutrient deficiencies and informing users of them.

[0010] A "deep learning algorithm" is a type of machine learning technique that uses multi-layered artificial neural networks to learn patterns from large amounts of data, enabling advanced processing such as image recognition.

[0011] "Means for identifying food components" refers to technologies that identify specific nutritional components contained in food and determine their type, quantity, etc.

[0012] "User's individual health data" refers to health-related information specific to the user, such as age, gender, lifestyle, and past health history. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system that enables users to consume nutritionally balanced meals on a daily basis. The following describes specific embodiments of this system.

[0035] First, users take pictures of their daily meals using a device such as a smartphone or tablet. The device then sends this image data to a server via an application. The server has high processing power in a cloud environment and can perform real-time image analysis.

[0036] The server uses a deep learning algorithm to analyze the received images and identify ingredients and dishes. This algorithm has been pre-trained on a rich dataset and is capable of pattern recognition of similar images.

[0037] Next, the server uses the analyzed food component data and the user's individual health data (including age, gender, and health history) to calculate which nutrients are deficient. This process is based on physiological and nutritional models.

[0038] Once the server identifies any nutrient deficiencies, it generates optimal meal suggestions to compensate for them. These suggestions include specific ingredients, cooking methods, and recommended meal timings. For example, if a vitamin C deficiency is detected, it might suggest recipes for orange smoothies or bell pepper salads.

[0039] The generated meal suggestions are sent from the server to the user's terminal, and the user can easily review the information. Based on these suggestions, the user can prepare their next meal and ensure nutritional balance.

[0040] This system allows users to improve their diet and maintain their health. The key feature of this invention is that it provides personalized suggestions tailored to the user's characteristics, achieving highly accurate nutritional management that cannot be obtained with conventional methods.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user takes a picture of their meal using a smartphone or tablet. The device then imports the resulting image directly into the application.

[0044] Step 2:

[0045] The device transmits the acquired meal image data to the server via the application. During this process, the image data is compressed or encrypted for secure and efficient transmission.

[0046] Step 3:

[0047] The server receives the transmitted image data and begins image analysis using a deep learning algorithm. It utilizes a pre-trained neural network model to accurately recognize ingredients and dishes within the image.

[0048] Step 4:

[0049] The server retrieves relevant food component data from a database based on the food ingredient information obtained through image analysis. This creates a detailed profile of the nutrients contained in the image.

[0050] Step 5:

[0051] The server integrates the user's individual health data (age, gender, health history, etc.) with food composition data to perform analysis to identify nutritional deficiencies. It then calculates and evaluates the user's current dietary status based on physiological models and nutritional theories.

[0052] Step 6:

[0053] The server generates meal suggestions to address identified nutrient deficiencies. These suggestions include recipes and cooking instructions using nutrient-rich ingredients. The suggestions are customized to take into account the user's preferences and allergy information.

[0054] Step 7:

[0055] The server sends the generated meal suggestions to the user's device. The user can review the suggestions displayed on their device and use them to prepare their next meal. This information supports the user in making decisions to maintain a healthy diet.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In today's busy lifestyle, it is difficult for users to easily manage the nutritional balance of their daily meals. In particular, making effective meal choices that take into account individual health conditions and nutrient deficiencies is difficult without specialized knowledge. There is a need to solve this problem.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for integrating the food component data and individual health data to generate and present meal suggestions to improve nutritional balance. This makes it possible for the user to easily select a nutritionally balanced meal that suits them and maintain their health.

[0061] A "user" refers to an individual who takes pictures of food and receives personalized nutritional management services based on that data.

[0062] "Food images" refer to image data of meals or ingredients taken by users.

[0063] "Food composition data" refers to information about the nutrients and components contained in food ingredients and dishes, obtained by analyzing food images.

[0064] "Individual health data" refers to personal health information related to nutrition management, such as the user's age, gender, and past medical history.

[0065] "Nutrients" refer to important substances found in food and dishes that contribute to maintaining human health and improving quality of life.

[0066] "Meal suggestions" refer to information that presents suitable ingredients and cooking options to the user, based on food composition data and individual health data.

[0067] A "server" refers to a device located in a cloud environment that possesses the computing resources and functions necessary for analyzing food images and managing nutritional information.

[0068] A "machine learning algorithm" refers to a computational method that allows computers to automatically learn patterns from data and analyze unknown data.

[0069] This system combines multiple technological elements to help users consume nutritionally balanced meals on a daily basis. Users take pictures of each meal using a smartphone or tablet. This allows users to easily collect and manage their meal data.

[0070] The device transmits captured image data to a server via a dedicated application. Typical hardware such as smartphones and tablets are expected to be used. The server possesses high computing power in a cloud environment and the ability to process image data in real time. Specifically, a commercial cloud service platform can be utilized.

[0071] The server analyzes the received images using machine learning algorithms based on deep learning. Neural network models such as ResNet and VGG are used in this process. Through image analysis, the server identifies ingredients and dishes and extracts corresponding food component data. This data forms a crucial foundation for nutritional analysis based on user-submitted images.

[0072] Next, the server integrates the user's individual health data with food ingredient data. Users register data such as age, gender, and past health history from their device beforehand. Based on this, the server identifies the nutrients the user needs and generates personalized meal suggestions. For example, for a user who is deficient in vitamin C, the server can suggest recipes for an orange smoothie or a bell pepper salad.

[0073] The generated meal suggestions are sent from the server to the terminal, allowing the user to easily review them within the application. The user can then prepare their next meal based on the suggestions and ensure a balanced diet. An example of a prompt might be, "Please suggest a suitable vitamin D-rich dish for dinner." This allows users to efficiently manage their eating habits and receive support in maintaining their health.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The user takes a picture of their meal using a smartphone or tablet. They launch the camera app and take a picture that shows the entire meal. The input image data is saved in a ready-to-use format. This serves as the basic data for subsequent analysis processes.

[0077] Step 2:

[0078] The terminal sends captured image data to the server via a dedicated application. Here, the application is responsible for transferring the image files to the server in the correct format. The input is the image data sent from the terminal, and the output is the image data received by the server.

[0079] Step 3:

[0080] The server analyzes the received images using deep learning. Specifically, the server uses neural network models such as ResNet and VGG to identify ingredients and dishes within the images. The input to this process is image data, and the output is food component data. The analysis results in a list of ingredient names and dish names.

[0081] Step 4:

[0082] The server integrates the food component data obtained from image analysis with the user's individual health data. At this time, the server utilizes information previously provided by the user, such as age, gender, and health history. The input consists of food component data and individual health data, which are used to identify deficient nutrients. The output of this step is a list of deficient nutrients.

[0083] Step 5:

[0084] The server generates meal suggestions to supplement any nutritional deficiencies. This process uses a nutritional model to determine specific ingredients, cooking methods, and timing of intake. The meal suggestions generated by the server are customized for each user and designed to ensure efficient intake of specific nutrients. The input is a list of deficient nutrients, and the output is a personalized meal suggestion.

[0085] Step 6:

[0086] Meal suggestions generated by the server are sent to the terminal, and the user checks the suggestions through the application. The user uses this information to prepare their next meal and improve their nutritional balance. The input is the meal suggestions sent from the server, and the output is the user's plan for their next meal. Specifically, the user is notified when meal suggestions arrive via a notification function.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Modern people lead busy lives, making it difficult to be mindful of nutritional balance in their daily meals. However, consuming a nutritionally balanced diet tailored to each individual is crucial for maintaining and improving health. Furthermore, choosing the optimal menu from many options is not easy. To solve these problems, there is a need for a means to accurately understand an individual's nutritional status and provide optimal nutrition without hassle.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for selecting and providing a service to the user that proposes a nutritionally balanced menu to supplement the identified deficient nutrients. This makes it possible for the user to easily select and consume meals that take nutritional balance into consideration.

[0092] "User-submitted food images" refer to visual information of meals recorded by users using camera-equipped devices.

[0093] "Food composition data" refers to information such as nutrients and calories of each ingredient extracted from analyzed food images.

[0094] "Individual health data" refers to information about a specific user's health status, such as age, gender, and past health history.

[0095] "Nutrient deficiencies" refer to essential nutrients that users are not adequately obtaining through their current diet, based on their individual health data.

[0096] A "nutritionally balanced menu" is a meal plan that includes appropriate nutrients, suggested to supplement any nutritional deficiencies the user may have.

[0097] "The means of selecting and providing services to users" refers to a function that selects the most suitable service from a large number of options and guides the user to it, with the aim of supplementing the nutrients that the user is lacking.

[0098] To implement this invention, the server begins processing upon receiving food images captured by the user's terminal. The user takes photos of their meals with a mobile device such as a smartphone or tablet and sends them to the server through an application. The server is located in a cloud environment with high-speed processing capabilities and quickly analyzes the received image data. The analysis uses a deep learning algorithm that has been trained on a rich dataset beforehand, which allows for accurate identification of ingredients and dishes.

[0099] Next, the server uses the food component data obtained through analysis to compare it with the user's individual health data and identify any nutritional deficiencies. This individual health data includes personal elements such as the user's age, gender, and health history, and a nutritional model is built based on this data.

[0100] Once a nutritional deficiency is identified, the server selects the most suitable service to supplement it. Specifically, it accesses available delivery services and suggests nutritionally balanced menus. These suggestions are sent to the user's device, allowing them to easily place an order with a touch.

[0101] This system allows users to easily select nutritionally balanced meals in their daily lives. For example, if a vitamin D deficiency is detected from a breakfast image, a menu including "salmon and spinach salad" may be recommended for lunch. A possible example of a specific prompt message would be, "Analyze this image I took for breakfast and suggest a nutritionally balanced menu for lunch."

[0102] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0103] Step 1:

[0104] The device takes a picture of the meal and sends the image data to the server via the application. The input here is an image of the meal, and the output is the transmission of image data to the server.

[0105] Step 2:

[0106] The server analyzes received image data in the cloud using a deep learning algorithm. Based on the image data as input, food component data is extracted through analysis, and ingredients and dishes are identified. The output is food component data.

[0107] Step 3:

[0108] The server combines analyzed food component data with the user's individual health data to identify deficient nutrients. The inputs are food component data and individual health data, and the output is a list of deficient nutrients.

[0109] Step 4:

[0110] The server suggests nutritionally balanced menu options to supplement any missing nutrients. The server uses menu information from delivery services as input and generates an optimal meal plan tailored to the user's situation. The output is a list of optimal menu plans.

[0111] Step 5:

[0112] The terminal displays suggested menus received from the server to the user, offering delivery order options. The input is the suggested menu, and the output is the meal plan displayed to the user. The user can review this and place an order directly.

[0113] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0114] This invention combines a system for improving a user's nutritional status with an emotion engine that recognizes the user's emotions. The embodiments of this system are described in detail below.

[0115] First, the user takes a picture of their meal using a device such as a smartphone or tablet. The device then sends this image to a server via an application. The server uses a deep learning algorithm to analyze the received image and identify the ingredients and dishes. This process identifies the nutrients contained in the meal.

[0116] Next, the server receives emotional data acquired from the user's terminal through the emotion engine. This emotion engine analyzes data to identify the user's emotional state in their daily life. This includes functions to read emotions from the user's voice, facial expressions, and text input.

[0117] The server comprehensively analyzes nutritional data obtained from ingredients, individual user health data (age, gender, health history, etc.), and emotional data. This allows for meal suggestions that consider not only nutrition but also the user's emotional state. For example, if a user is feeling stressed, recipes using ingredients that are expected to have a relaxing effect will be suggested.

[0118] Meal suggestions are generated by the server and sent to the user's device. These suggestions include detailed recipes, a list of necessary ingredients, cooking instructions, and even tips for emotional well-being. Users can use this information to prepare meals, resulting in both physical nutritional balance and psychological satisfaction.

[0119] Thus, by adding an emotional element to nutritional management, the present invention realizes a user experience that was not possible with conventional meal management systems. Through this system, users can lead a healthier and more comfortable life.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The user takes a picture of their meal using a smartphone or tablet. The device then imports the captured image data into the application in real time and prepares it.

[0123] Step 2:

[0124] The device sends the captured image to the server. This communication uses an encryption protocol to efficiently and securely transfer the image data.

[0125] Step 3:

[0126] The server analyzes the received images using deep learning algorithms to identify the ingredients and components of the food contained within the images. This reveals information about the nutrients contained in the meal.

[0127] Step 4:

[0128] The server also receives user emotion data collected from the terminal via the emotion engine. This data indicates the emotional state estimated from the user's tone of voice, facial expressions, input patterns, etc.

[0129] Step 5:

[0130] The server integrates and comprehensively analyzes nutritional data from ingredients, individual user health data (e.g., age, gender, health history), and emotional data. This analysis provides a comprehensive understanding of the user's health and emotional state, identifying deficient nutrients and emotional states that need improvement.

[0131] Step 6:

[0132] Based on the extracted information, the server generates meal suggestions that take into account the user's emotional state. These suggestions include ingredients and recipes that are expected to have a balanced nutritional profile and help stabilize emotions.

[0133] Step 7:

[0134] The generated meal suggestions are sent from the server to the terminal. Users can review these suggestions on their terminal and prepare their next meal using the suggested recipes and ingredients.

[0135] This series of steps allows users to easily select and implement meals that are tailored to their individual health and emotional state.

[0136] (Example 2)

[0137] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0138] In modern society, individual users are required to maintain nutritional balance amidst diverse eating habits and living environments. However, conventional systems have struggled to provide meal suggestions that take into account the user's emotional state. This invention aims to solve the problem that conventional nutrition management systems cannot integrate the user's psychological state and nutritional status when making suggestions.

[0139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0140] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health information; and means for generating and presenting meal suggestions to supplement the identified deficient nutrients. This makes it possible to provide more appropriate nutritional suggestions that take into account the emotional state of each individual user.

[0141] "Food images" are image data taken by users for the purpose of recording their meals.

[0142] "Food composition data" refers to data that contains information about nutrients and ingredients identified through the analysis of food images.

[0143] "Individual health information" refers to information about each user's unique health status and history, including age, gender, and past health history.

[0144] "Nutrient deficiencies" refer to nutrients that are below the required amount, as determined based on the user's individual health information.

[0145] "Emotional data" refers to information that represents the user's psychological state, and includes data obtained from voice, facial expressions, text input, and other sources.

[0146] A "machine learning algorithm" is a computational method used in data analysis, aimed at identifying specific patterns or features.

[0147] This invention is a system that comprehensively analyzes a user's nutritional and emotional status and provides personalized meal suggestions. The main components of the system include image analysis, nutrient identification, emotional state analysis, and meal suggestion generation.

[0148] Users take pictures of their daily meals using devices such as smartphones and tablets. This image data is sent from the device to a server via the internet. The image data sent by the device is processed on the server.

[0149] The server analyzes the received images using machine learning algorithms, specifically deep learning technologies such as TENSORFLOW® and PyTorch. This allows the ingredients and dishes identified from the images to be compared with a nutrient database, and the nutrients are identified.

[0150] In addition, users utilize the device's emotion engine to collect emotional data in their daily lives. This engine uses speech analysis, image recognition, and text input analysis (for example, using Google® Cloud Speech-to-Text and Microsoft® Azure® Face API) to evaluate the user's psychological state. The evaluated emotional data is sent from the device to a server.

[0151] The server uses a generative AI model to perform a comprehensive analysis using nutrient data, emotional data, and the user's individual health information (age, gender, health history, etc.). The generative AI model uses the prompt message, "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress."

[0152] For example, if a user is feeling stressed, this system can suggest recipes using ingredients known to reduce stress. The meal suggestions generated by the server are sent to the user's device, where they receive detailed recipes, ingredient lists, and cooking instructions, allowing them to prepare meals that take their health and emotional state into consideration. Through this system, users can expect to achieve both physical health and psychological satisfaction simultaneously.

[0153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0154] Step 1:

[0155] Users take pictures of their daily meals using their smartphones or tablets. These images serve as input data. The device automatically sends these images to the server. Specifically, the image data is compressed before transmission to improve communication efficiency.

[0156] Step 2:

[0157] The server analyzes image data received from the terminal using a deep learning algorithm. It converts the input image into features and performs calculations to identify specific ingredients and dishes. The analysis results output the identified ingredients and their nutritional information. This process utilizes a model based on TensorFlow.

[0158] Step 3:

[0159] The user's device routinely collects emotional data. This includes features that detect emotions from voice, facial expressions, and text input, which are treated as input data. The device uses an emotion engine to analyze the data and perform calculations to identify the user's psychological state. The analyzed emotional data is sent to a server as output. Google Cloud Speech-to-Text and Microsoft Azure Face APIs are utilized in this process.

[0160] Step 4:

[0161] The server integrates and analyzes nutrient data, the user's individual health information (including age, gender, and health history), and emotional data. This integrated data is used as input, and the prompt message "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress." is input to the generating AI model. This process performs calculations to generate appropriate meal suggestions based on the integrated data. The server then receives the resulting suggestions as output.

[0162] Step 5:

[0163] The server sends the generated meal suggestions to the user's device. The output meal suggestions include detailed recipes, a list of necessary ingredients, and cooking instructions. The user receives this information and uses it to plan their daily meals. Based on this information, the user can prepare meals that are optimal for their nutritional needs and emotional state.

[0164] (Application Example 2)

[0165] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0166] In modern society, maintaining individual health is becoming increasingly important, but conventional nutrition management systems have the challenge of not being able to suggest meals that take into account the user's emotional state. Furthermore, while it would be desirable to use autonomous robotic devices to efficiently manage individual meals at home, this is not currently possible with existing systems.

[0167] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0168] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; means for generating and presenting meal suggestions to supplement the identified deficient nutrients; means for analyzing the user's emotional state and customizing meal suggestions based on emotional data; and means for a robotic device to autonomously collect the user's emotions during meals and adjust the environment or provide advice based on the analysis results. This enables meal suggestions that comprehensively consider the user's health state and emotions, realizing effective meal management within the home.

[0169] "Food images" are image data taken by users to record the contents of their meals, and they serve as basic data for identifying food components.

[0170] "Food composition data" refers to information about nutrients and compounds contained in food ingredients and dishes, obtained by analyzing food images.

[0171] "Individual health data" refers to data that indicates the health status of an individual user, and includes information such as age, gender, and past health history.

[0172] "Emotional state" refers to the type and intensity of emotions a user experiences in their daily life or in specific situations, and is determined from factors such as voice and facial expressions.

[0173] A "robot device" is an autonomous electronic device intended to provide user support within the home, and has the function of collecting and analyzing emotional and environmental data to provide a user interface.

[0174] "Meal suggestions" refer to information about recommended meal menus and nutritional supplements that are generated taking into account the user's health condition and emotional state.

[0175] "Adjusting the environment" refers to the robotic device changing surrounding environmental elements in response to the user's emotional state, including, for example, the color of the lighting or the selection of music.

[0176] In this invention, a user takes a picture of their meal using a terminal and sends the image to a server. The server uses TensorFlow to analyze the received image and identify food component data. Because this analysis uses a deep learning algorithm, various ingredients and dishes can be accurately identified.

[0177] Next, the server references the user's individual health data to identify any nutritional deficiencies. This takes into account age, gender, and past health history. Furthermore, the user's device or robotic device uses an emotion engine to analyze the user's emotional state from their voice and facial expressions. This emotional data is analyzed using OpenCV for facial expression analysis and processed using natural language processing libraries for voice data.

[0178] The analyzed information is comprehensively analyzed by the server to generate meal suggestions that take into account the user's health and emotional state. These suggestions include recipes for dishes containing specific nutrients to stabilize emotions, as well as advice on adjusting the environment. For example, it might recommend herbal teas that are expected to have a relaxing effect, or playing calming music.

[0179] Furthermore, consumer robots autonomously provide assistance within the home in response to changes in the user's daily emotions and health. Through interaction with the user, robotic devices can provide an optimal environment. This system is expected to enable users to effectively manage both their physical health and psychological well-being.

[0180] For example, if a user is feeling stressed, the robot might suggest, "Why don't you try a relaxing herbal tea today? I'll dim the lights a little."

[0181] The prompt message to the generating AI model will be in the following format: "Suggest a relaxing meal menu for the user, especially if they are feeling stressed."

[0182] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0183] Step 1:

[0184] The user takes a picture of their meal using their device. This image is sent to the server. The input is the image data captured by the user, and the output is the image file to be sent to the server.

[0185] Step 2:

[0186] The server processes the received images and uses TensorFlow to leverage deep learning algorithms to analyze food components. The input is image data, and the output is identified food component data. Specifically, it converts the ingredients and dishes in the image into feature vectors and performs classification using a pre-trained model.

[0187] Step 3:

[0188] The server uses the analyzed food component data to compare it with the user's individual health data and identify any nutritional deficiencies. Input requires food component data and the user's health data, and output is a list of deficient nutrients. Data matching is performed using a database.

[0189] Step 4:

[0190] The user's device or robotic device analyzes emotional states from voice and facial expressions. Facial expression analysis is performed using OpenCV, and voice data is processed using a natural language processing library. Input is facial expression data (voice and images), and output is the user's emotion determination result.

[0191] Step 5:

[0192] The server comprehensively analyzes food component data, health data, nutrient deficiencies, and emotional data to generate meal suggestions tailored to the user. The input consists of various data, and the output is detailed information about the meal suggestions presented to the user. A generation AI model is used to provide menu suggestions that reflect individual factors to the greatest extent possible.

[0193] Step 6:

[0194] The robotic device communicates meal suggestions to the user and adjusts the environment as needed. Input is meal suggestion information from a server, and output is voice prompts and instructions for environmental adjustments. Specifically, it uses speech synthesis technology to communicate the suggestions verbally and changes environmental factors such as lighting color.

[0195] Step 7:

[0196] Users implement suggestions and enjoy their meals. This makes it possible to maintain physical health and psychological well-being.

[0197] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0198] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0199] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0200] [Second Embodiment]

[0201] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0202] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0203] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0204] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0205] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0206] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0207] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0208] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0209] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0210] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0211] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0212] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0213] This invention provides a system that enables users to consume nutritionally balanced meals on a daily basis. The following describes specific embodiments of this system.

[0214] First, users take pictures of their daily meals using a device such as a smartphone or tablet. The device then sends this image data to a server via an application. The server has high processing power in a cloud environment and can perform real-time image analysis.

[0215] The server uses a deep learning algorithm to analyze the received images and identify ingredients and dishes. This algorithm has been pre-trained on a rich dataset and is capable of pattern recognition of similar images.

[0216] Next, the server uses the analyzed food component data and the user's individual health data (including age, gender, and health history) to calculate which nutrients are deficient. This process is based on physiological and nutritional models.

[0217] Once the server identifies any nutrient deficiencies, it generates optimal meal suggestions to compensate for them. These suggestions include specific ingredients, cooking methods, and recommended meal timings. For example, if a vitamin C deficiency is detected, it might suggest recipes for orange smoothies or bell pepper salads.

[0218] The generated meal suggestions are sent from the server to the user's terminal, and the user can easily review the information. Based on these suggestions, the user can prepare their next meal and ensure nutritional balance.

[0219] This system allows users to improve their diet and maintain their health. The key feature of this invention is that it provides personalized suggestions tailored to the user's characteristics, achieving highly accurate nutritional management that cannot be obtained with conventional methods.

[0220] The following describes the processing flow.

[0221] Step 1:

[0222] The user takes a picture of their meal using a smartphone or tablet. The device then imports the resulting image directly into the application.

[0223] Step 2:

[0224] The device transmits the acquired meal image data to the server via the application. During this process, the image data is compressed or encrypted for secure and efficient transmission.

[0225] Step 3:

[0226] The server receives the transmitted image data and begins image analysis using a deep learning algorithm. It utilizes a pre-trained neural network model to accurately recognize ingredients and dishes within the image.

[0227] Step 4:

[0228] The server retrieves relevant food component data from a database based on the food ingredient information obtained through image analysis. This creates a detailed profile of the nutrients contained in the image.

[0229] Step 5:

[0230] The server integrates the user's individual health data (age, gender, health history, etc.) with food composition data to perform analysis to identify nutritional deficiencies. It then calculates and evaluates the user's current dietary status based on physiological models and nutritional theories.

[0231] Step 6:

[0232] The server generates meal suggestions to address identified nutrient deficiencies. These suggestions include recipes and cooking instructions using nutrient-rich ingredients. The suggestions are customized to take into account the user's preferences and allergy information.

[0233] Step 7:

[0234] The server sends the generated meal suggestions to the user's device. The user can review the suggestions displayed on their device and use them to prepare their next meal. This information supports the user in making decisions to maintain a healthy diet.

[0235] (Example 1)

[0236] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] In today's busy lifestyle, it is difficult for users to easily manage the nutritional balance of their daily meals. In particular, making effective meal choices that take into account individual health conditions and nutrient deficiencies is difficult without specialized knowledge. There is a need to solve this problem.

[0238] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0239] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for integrating the food component data and individual health data to generate and present meal suggestions to improve nutritional balance. This makes it possible for the user to easily select a nutritionally balanced meal that suits them and maintain their health.

[0240] A "user" refers to an individual who takes pictures of food and receives personalized nutritional management services based on that data.

[0241] "Food images" refer to image data of meals or ingredients taken by users.

[0242] "Food composition data" refers to information about the nutrients and components contained in food ingredients and dishes, obtained by analyzing food images.

[0243] "Individual health data" refers to personal health information related to nutrition management, such as the user's age, gender, and past medical history.

[0244] "Nutrients" refer to important substances found in food and dishes that contribute to maintaining human health and improving quality of life.

[0245] "Meal suggestions" refer to information that presents suitable ingredients and cooking options to the user, based on food composition data and individual health data.

[0246] A "server" refers to a device located in a cloud environment that possesses the computing resources and functions necessary for analyzing food images and managing nutritional information.

[0247] A "machine learning algorithm" refers to a computational method that allows computers to automatically learn patterns from data and analyze unknown data.

[0248] This system combines multiple technological elements to help users consume nutritionally balanced meals on a daily basis. Users take pictures of each meal using a smartphone or tablet. This allows users to easily collect and manage their meal data.

[0249] The device transmits captured image data to a server via a dedicated application. Typical hardware such as smartphones and tablets are expected to be used. The server possesses high computing power in a cloud environment and the ability to process image data in real time. Specifically, a commercial cloud service platform can be utilized.

[0250] The server analyzes the received images using machine learning algorithms based on deep learning. Neural network models such as ResNet and VGG are used in this process. Through image analysis, the server identifies ingredients and dishes and extracts corresponding food component data. This data forms a crucial foundation for nutritional analysis based on user-submitted images.

[0251] Next, the server integrates the user's individual health data with food ingredient data. Users register data such as age, gender, and past health history from their device beforehand. Based on this, the server identifies the nutrients the user needs and generates personalized meal suggestions. For example, for a user who is deficient in vitamin C, the server can suggest recipes for an orange smoothie or a bell pepper salad.

[0252] The generated meal suggestions are sent from the server to the terminal, allowing the user to easily review them within the application. The user can then prepare their next meal based on the suggestions and ensure a balanced diet. An example of a prompt might be, "Please suggest a suitable vitamin D-rich dish for dinner." This allows users to efficiently manage their eating habits and receive support in maintaining their health.

[0253] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0254] Step 1:

[0255] The user takes a picture of their meal using a smartphone or tablet. They launch the camera app and take a picture that shows the entire meal. The input image data is saved in a ready-to-use format. This serves as the basic data for subsequent analysis processes.

[0256] Step 2:

[0257] The terminal sends captured image data to the server via a dedicated application. Here, the application is responsible for transferring the image files to the server in the correct format. The input is the image data sent from the terminal, and the output is the image data received by the server.

[0258] Step 3:

[0259] The server analyzes the received images using deep learning. Specifically, the server uses neural network models such as ResNet and VGG to identify ingredients and dishes within the images. The input to this process is image data, and the output is food component data. The analysis results in a list of ingredient names and dish names.

[0260] Step 4:

[0261] The server integrates the food component data obtained from image analysis with the user's individual health data. At this time, the server utilizes information previously provided by the user, such as age, gender, and health history. The input consists of food component data and individual health data, which are used to identify deficient nutrients. The output of this step is a list of deficient nutrients.

[0262] Step 5:

[0263] The server generates meal suggestions to supplement any nutritional deficiencies. This process uses a nutritional model to determine specific ingredients, cooking methods, and timing of intake. The meal suggestions generated by the server are customized for each user and designed to ensure efficient intake of specific nutrients. The input is a list of deficient nutrients, and the output is a personalized meal suggestion.

[0264] Step 6:

[0265] Meal suggestions generated by the server are sent to the terminal, and the user checks the suggestions through the application. The user uses this information to prepare their next meal and improve their nutritional balance. The input is the meal suggestions sent from the server, and the output is the user's plan for their next meal. Specifically, the user is notified when meal suggestions arrive via a notification function.

[0266] (Application Example 1)

[0267] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0268] Modern people lead busy lives, making it difficult to be mindful of nutritional balance in their daily meals. However, consuming a nutritionally balanced diet tailored to each individual is crucial for maintaining and improving health. Furthermore, choosing the optimal menu from many options is not easy. To solve these problems, there is a need for a means to accurately understand an individual's nutritional status and provide optimal nutrition without hassle.

[0269] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0270] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for selecting and providing a service to the user that proposes a nutritionally balanced menu to supplement the identified deficient nutrients. This makes it possible for the user to easily select and consume meals that take nutritional balance into consideration.

[0271] "User-submitted food images" refer to visual information of meals recorded by users using camera-equipped devices.

[0272] "Food composition data" refers to information such as nutrients and calories of each ingredient extracted from analyzed food images.

[0273] "Individual health data" refers to information about a specific user's health status, such as age, gender, and past health history.

[0274] "Nutrient deficiencies" refer to essential nutrients that users are not adequately obtaining through their current diet, based on their individual health data.

[0275] A "nutritionally balanced menu" is a meal plan that includes appropriate nutrients, suggested to supplement any nutritional deficiencies the user may have.

[0276] "The means of selecting and providing services to users" refers to a function that selects the most suitable service from a large number of options and guides the user to it, with the aim of supplementing the nutrients that the user is lacking.

[0277] To implement this invention, the server begins processing upon receiving food images captured by the user's terminal. The user takes photos of their meals with a mobile device such as a smartphone or tablet and sends them to the server through an application. The server is located in a cloud environment with high-speed processing capabilities and quickly analyzes the received image data. The analysis uses a deep learning algorithm that has been trained on a rich dataset beforehand, which allows for accurate identification of ingredients and dishes.

[0278] Next, the server uses the food component data obtained through analysis to compare it with the user's individual health data and identify any nutritional deficiencies. This individual health data includes personal elements such as the user's age, gender, and health history, and a nutritional model is built based on this data.

[0279] Once a nutritional deficiency is identified, the server selects the most suitable service to supplement it. Specifically, it accesses available delivery services and suggests nutritionally balanced menus. These suggestions are sent to the user's device, allowing them to easily place an order with a touch.

[0280] With this system, users can easily select meals that take into account nutritional balance in their daily lives. For example, if a lack of vitamin D is detected from an image of breakfast, a menu including "salmon and spinach salad" for lunch may be recommended. As an example of a specific prompt sentence, an input such as "Analyze this image taken at breakfast and propose a recommended menu with a good nutritional balance for lunch." can be considered.

[0281] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0282] Step 1:

[0283] The terminal takes a picture of the meal image and sends the image data to the server through the application. The input here is the image of the meal content, and the output is the transmission of the image data to the server.

[0284] Step 2:

[0285] The server analyzes the received image data using a deep learning algorithm on the cloud. Based on the image data as the input, food component data is extracted by the analysis to identify ingredients and dishes. The output is the food component data.

[0286] Step 3:

[0287] The server combines the analyzed food component data and the user's individual health data to identify the nutrients that are lacking. The inputs are the food component data and the individual health data, and as the output, a list of lacking nutrients is obtained.

[0288] Step 4:

[0289] The server proposes a nutritionally balanced menu as an option to supplement the lacking nutrients. The server takes the menu information of the delivery service as the input and generates an optimal meal plan suitable for the user's situation. The output is the list of optimal menu plans.

[0290] Step 5:

[0291] The terminal displays suggested menus received from the server to the user, offering delivery order options. The input is the suggested menu, and the output is the meal plan displayed to the user. The user can review this and place an order directly.

[0292] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0293] This invention combines a system for improving a user's nutritional status with an emotion engine that recognizes the user's emotions. The embodiments of this system are described in detail below.

[0294] First, the user takes a picture of their meal using a device such as a smartphone or tablet. The device then sends this image to a server via an application. The server uses a deep learning algorithm to analyze the received image and identify the ingredients and dishes. This process identifies the nutrients contained in the meal.

[0295] Next, the server receives emotional data acquired from the user's terminal through the emotion engine. This emotion engine analyzes data to identify the user's emotional state in their daily life. This includes functions to read emotions from the user's voice, facial expressions, and text input.

[0296] The server comprehensively analyzes nutritional data obtained from ingredients, individual user health data (age, gender, health history, etc.), and emotional data. This allows for meal suggestions that consider not only nutrition but also the user's emotional state. For example, if a user is feeling stressed, recipes using ingredients that are expected to have a relaxing effect will be suggested.

[0297] Meal suggestions are generated by the server and sent to the user's device. These suggestions include detailed recipes, a list of necessary ingredients, cooking instructions, and even tips for emotional well-being. Users can use this information to prepare meals, resulting in both physical nutritional balance and psychological satisfaction.

[0298] Thus, by adding an emotional element to nutritional management, the present invention realizes a user experience that was not possible with conventional meal management systems. Through this system, users can lead a healthier and more comfortable life.

[0299] The following describes the processing flow.

[0300] Step 1:

[0301] The user takes a picture of their meal using a smartphone or tablet. The device then imports the captured image data into the application in real time and prepares it.

[0302] Step 2:

[0303] The device sends the captured image to the server. This communication uses an encryption protocol to efficiently and securely transfer the image data.

[0304] Step 3:

[0305] The server analyzes the received images using deep learning algorithms to identify the ingredients and components of the food contained within the images. This reveals information about the nutrients contained in the meal.

[0306] Step 4:

[0307] The server also receives the user's emotion data collected from the terminal via the emotion engine. This data indicates the emotional state estimated from the user's voice tone, facial expression, input pattern, etc.

[0308] Step 5:

[0309] The server integrates and comprehensively analyzes the nutritional data of the food ingredients, the user's individual health data (e.g., age, gender, health history), and the emotion data. Through this analysis, the server comprehensively grasps the user's health state and emotional state, and extracts the lacking nutrients and the emotional state to be improved.

[0310] Step 6:

[0311] Based on the extracted information, the server generates a meal recommendation considering the user's emotional state. This recommendation includes food ingredients and recipes that can balance nutrition and are expected to have an effect on stabilizing emotions.

[0312] Step 7:

[0313] The generated meal recommendation is sent from the server to the terminal. The user can check these recommendations on the terminal and prepare the next meal using the recommended recipes and food ingredients.

[0314] Through this series of steps, the user can easily select and implement a meal suitable for their individual health state and emotional state.

[0315] (Example 2)

[0316] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0317] In modern society, individual users are required to maintain nutritional balance amidst diverse eating habits and living environments. However, conventional systems have struggled to provide meal suggestions that take into account the user's emotional state. This invention aims to solve the problem that conventional nutrition management systems cannot integrate the user's psychological state and nutritional status when making suggestions.

[0318] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0319] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health information; and means for generating and presenting meal suggestions to supplement the identified deficient nutrients. This makes it possible to provide more appropriate nutritional suggestions that take into account the emotional state of each individual user.

[0320] "Food images" are image data taken by users for the purpose of recording their meals.

[0321] "Food composition data" refers to data that contains information about nutrients and ingredients identified through the analysis of food images.

[0322] "Individual health information" refers to information about each user's unique health status and history, including age, gender, and past health history.

[0323] "Nutrient deficiencies" refer to nutrients that are below the required amount, as determined based on the user's individual health information.

[0324] "Emotional data" refers to information that represents the user's psychological state, and includes data obtained from voice, facial expressions, text input, and other sources.

[0325] A "machine learning algorithm" is a computational method used in data analysis, aimed at identifying specific patterns or features.

[0326] This invention is a system that comprehensively analyzes a user's nutritional and emotional status and provides personalized meal suggestions. The main components of the system include image analysis, nutrient identification, emotional state analysis, and meal suggestion generation.

[0327] Users take pictures of their daily meals using devices such as smartphones and tablets. This image data is sent from the device to a server via the internet. The image data sent by the device is processed on the server.

[0328] The server analyzes the received images using machine learning algorithms, specifically deep learning technologies such as TensorFlow and PyTorch. This allows the ingredients and dishes identified from the images to be compared with a nutrient database, and their nutrients are identified.

[0329] In addition, users utilize the device's emotion engine to collect emotional data in their daily lives. This engine leverages speech analysis, image recognition, and text input analysis (e.g., using Google Cloud Speech-to-Text or Microsoft Azure Face API) to assess the user's psychological state. The assessed emotional data is then sent from the device to a server.

[0330] The server uses a generative AI model to perform a comprehensive analysis using nutrient data, emotional data, and the user's individual health information (age, gender, health history, etc.). The generative AI model uses the prompt message, "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress."

[0331] For example, if a user is feeling stressed, this system can suggest recipes using ingredients known to reduce stress. The meal suggestions generated by the server are sent to the user's device, where they receive detailed recipes, ingredient lists, and cooking instructions, allowing them to prepare meals that take their health and emotional state into consideration. Through this system, users can expect to achieve both physical health and psychological satisfaction simultaneously.

[0332] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0333] Step 1:

[0334] Users take pictures of their daily meals using their smartphones or tablets. These images serve as input data. The device automatically sends these images to the server. Specifically, the image data is compressed before transmission to improve communication efficiency.

[0335] Step 2:

[0336] The server analyzes image data received from the terminal using a deep learning algorithm. It converts the input image into features and performs calculations to identify specific ingredients and dishes. The analysis results output the identified ingredients and their nutritional information. This process utilizes a model based on TensorFlow.

[0337] Step 3:

[0338] The user's device routinely collects emotional data. This includes features that detect emotions from voice, facial expressions, and text input, which are treated as input data. The device uses an emotion engine to analyze the data and perform calculations to identify the user's psychological state. The analyzed emotional data is sent to a server as output. Google Cloud Speech-to-Text and Microsoft Azure Face APIs are utilized in this process.

[0339] Step 4:

[0340] The server integrates and analyzes nutrient data, the user's individual health information (including age, gender, and health history), and emotional data. This integrated data is used as input, and the prompt message "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress." is input to the generating AI model. This process performs calculations to generate appropriate meal suggestions based on the integrated data. The server then receives the resulting suggestions as output.

[0341] Step 5:

[0342] The server sends the generated meal suggestions to the user's device. The output meal suggestions include detailed recipes, a list of necessary ingredients, and cooking instructions. The user receives this information and uses it to plan their daily meals. Based on this information, the user can prepare meals that are optimal for their nutritional needs and emotional state.

[0343] (Application Example 2)

[0344] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0345] In modern society, maintaining individual health is becoming increasingly important, but conventional nutrition management systems have the challenge of not being able to suggest meals that take into account the user's emotional state. Furthermore, while it would be desirable to use autonomous robotic devices to efficiently manage individual meals at home, this is not currently possible with existing systems.

[0346] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0347] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; means for generating and presenting meal suggestions to supplement the identified deficient nutrients; means for analyzing the user's emotional state and customizing meal suggestions based on emotional data; and means for a robotic device to autonomously collect the user's emotions during meals and adjust the environment or provide advice based on the analysis results. This enables meal suggestions that comprehensively consider the user's health state and emotions, realizing effective meal management within the home.

[0348] "Food images" are image data taken by users to record the contents of their meals, and they serve as basic data for identifying food components.

[0349] "Food composition data" refers to information about nutrients and compounds contained in food ingredients and dishes, obtained by analyzing food images.

[0350] "Individual health data" refers to data that indicates the health status of an individual user, and includes information such as age, gender, and past health history.

[0351] "Emotional state" refers to the type and intensity of emotions a user experiences in their daily life or in specific situations, and is determined from factors such as voice and facial expressions.

[0352] A "robot device" is an autonomous electronic device intended to provide user support within the home, and has the function of collecting and analyzing emotional and environmental data to provide a user interface.

[0353] "Meal suggestions" refer to information about recommended meal menus and nutritional supplements that are generated taking into account the user's health condition and emotional state.

[0354] "Adjusting the environment" refers to the robotic device changing surrounding environmental elements in response to the user's emotional state, including, for example, the color of the lighting or the selection of music.

[0355] In this invention, a user takes a picture of their meal using a terminal and sends the image to a server. The server uses TensorFlow to analyze the received image and identify food component data. Because this analysis uses a deep learning algorithm, various ingredients and dishes can be accurately identified.

[0356] Next, the server references the user's individual health data to identify any nutritional deficiencies. This takes into account age, gender, and past health history. Furthermore, the user's device or robotic device uses an emotion engine to analyze the user's emotional state from their voice and facial expressions. This emotional data is analyzed using OpenCV for facial expression analysis and processed using natural language processing libraries for voice data.

[0357] The analyzed information is comprehensively analyzed by the server to generate meal suggestions that take into account the user's health and emotional state. These suggestions include recipes for dishes containing specific nutrients to stabilize emotions, as well as advice on adjusting the environment. For example, it might recommend herbal teas that are expected to have a relaxing effect, or playing calming music.

[0358] Furthermore, consumer robots autonomously provide assistance within the home in response to changes in the user's daily emotions and health. Through interaction with the user, robotic devices can provide an optimal environment. This system is expected to enable users to effectively manage both their physical health and psychological well-being.

[0359] For example, if a user is feeling stressed, the robot might suggest, "Why don't you try a relaxing herbal tea today? I'll dim the lights a little."

[0360] The prompt message to the generating AI model will be in the following format: "Suggest a relaxing meal menu for the user, especially if they are feeling stressed."

[0361] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0362] Step 1:

[0363] The user takes a picture of their meal using their device. This image is sent to the server. The input is the image data captured by the user, and the output is the image file to be sent to the server.

[0364] Step 2:

[0365] The server processes the received images and uses TensorFlow to leverage deep learning algorithms to analyze food components. The input is image data, and the output is identified food component data. Specifically, it converts the ingredients and dishes in the image into feature vectors and performs classification using a pre-trained model.

[0366] Step 3:

[0367] The server uses the analyzed food component data to compare it with the user's individual health data and identify any nutritional deficiencies. Input requires food component data and the user's health data, and output is a list of deficient nutrients. Data matching is performed using a database.

[0368] Step 4:

[0369] The user's device or robotic device analyzes emotional states from voice and facial expressions. Facial expression analysis is performed using OpenCV, and voice data is processed using a natural language processing library. Input is facial expression data (voice and images), and output is the user's emotion determination result.

[0370] Step 5:

[0371] The server comprehensively analyzes food component data, health data, nutrient deficiencies, and emotional data to generate meal suggestions tailored to the user. The input consists of various data, and the output is detailed information about the meal suggestions presented to the user. A generation AI model is used to provide menu suggestions that reflect individual factors to the greatest extent possible.

[0372] Step 6:

[0373] The robotic device communicates meal suggestions to the user and adjusts the environment as needed. Input is meal suggestion information from a server, and output is voice prompts and instructions for environmental adjustments. Specifically, it uses speech synthesis technology to communicate the suggestions verbally and changes environmental factors such as lighting color.

[0374] Step 7:

[0375] Users implement suggestions and enjoy their meals. This makes it possible to maintain physical health and psychological well-being.

[0376] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0377] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0378] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0379] [Third Embodiment]

[0380] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0381] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0382] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0383] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0384] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0385] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0386] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0387] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0388] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0389] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0390] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0391] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0392] This invention provides a system that enables users to consume nutritionally balanced meals on a daily basis. The following describes specific embodiments of this system.

[0393] First, users take pictures of their daily meals using a device such as a smartphone or tablet. The device then sends this image data to a server via an application. The server has high processing power in a cloud environment and can perform real-time image analysis.

[0394] The server uses a deep learning algorithm to analyze the received images and identify ingredients and dishes. This algorithm has been pre-trained on a rich dataset and is capable of pattern recognition of similar images.

[0395] Next, the server uses the analyzed food component data and the user's individual health data (including age, gender, and health history) to calculate which nutrients are deficient. This process is based on physiological and nutritional models.

[0396] Once the server identifies any nutrient deficiencies, it generates optimal meal suggestions to compensate for them. These suggestions include specific ingredients, cooking methods, and recommended meal timings. For example, if a vitamin C deficiency is detected, it might suggest recipes for orange smoothies or bell pepper salads.

[0397] The generated meal suggestions are sent from the server to the user's terminal, and the user can easily review the information. Based on these suggestions, the user can prepare their next meal and ensure nutritional balance.

[0398] This system allows users to improve their diet and maintain their health. The key feature of this invention is that it provides personalized suggestions tailored to the user's characteristics, achieving highly accurate nutritional management that cannot be obtained with conventional methods.

[0399] The following describes the processing flow.

[0400] Step 1:

[0401] The user takes a picture of their meal using a smartphone or tablet. The device then imports the resulting image directly into the application.

[0402] Step 2:

[0403] The device transmits the acquired meal image data to the server via the application. During this process, the image data is compressed or encrypted for secure and efficient transmission.

[0404] Step 3:

[0405] The server receives the transmitted image data and begins image analysis using a deep learning algorithm. It utilizes a pre-trained neural network model to accurately recognize ingredients and dishes within the image.

[0406] Step 4:

[0407] The server retrieves relevant food component data from a database based on the food ingredient information obtained through image analysis. This creates a detailed profile of the nutrients contained in the image.

[0408] Step 5:

[0409] The server integrates the user's individual health data (age, gender, health history, etc.) with food composition data to perform analysis to identify nutritional deficiencies. It then calculates and evaluates the user's current dietary status based on physiological models and nutritional theories.

[0410] Step 6:

[0411] The server generates meal suggestions to address identified nutrient deficiencies. These suggestions include recipes and cooking instructions using nutrient-rich ingredients. The suggestions are customized to take into account the user's preferences and allergy information.

[0412] Step 7:

[0413] The server sends the generated meal suggestions to the user's device. The user can review the suggestions displayed on their device and use them to prepare their next meal. This information supports the user in making decisions to maintain a healthy diet.

[0414] (Example 1)

[0415] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0416] In today's busy lifestyle, it is difficult for users to easily manage the nutritional balance of their daily meals. In particular, making effective meal choices that take into account individual health conditions and nutrient deficiencies is difficult without specialized knowledge. There is a need to solve this problem.

[0417] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0418] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for integrating the food component data and individual health data to generate and present meal suggestions to improve nutritional balance. This makes it possible for the user to easily select a nutritionally balanced meal that suits them and maintain their health.

[0419] A "user" refers to an individual who takes pictures of food and receives personalized nutritional management services based on that data.

[0420] "Food images" refer to image data of meals or ingredients taken by users.

[0421] "Food composition data" refers to information about the nutrients and components contained in food ingredients and dishes, obtained by analyzing food images.

[0422] "Individual health data" refers to personal health information related to nutrition management, such as the user's age, gender, and past medical history.

[0423] "Nutrients" refer to important substances found in food and dishes that contribute to maintaining human health and improving quality of life.

[0424] "Meal suggestions" refer to information that presents suitable ingredients and cooking options to the user, based on food composition data and individual health data.

[0425] A "server" refers to a device located in a cloud environment that possesses the computing resources and functions necessary for analyzing food images and managing nutritional information.

[0426] A "machine learning algorithm" refers to a computational method that allows computers to automatically learn patterns from data and analyze unknown data.

[0427] This system combines multiple technological elements to help users consume nutritionally balanced meals on a daily basis. Users take pictures of each meal using a smartphone or tablet. This allows users to easily collect and manage their meal data.

[0428] The device transmits captured image data to a server via a dedicated application. Typical hardware such as smartphones and tablets are expected to be used. The server possesses high computing power in a cloud environment and the ability to process image data in real time. Specifically, a commercial cloud service platform can be utilized.

[0429] The server analyzes the received images using machine learning algorithms based on deep learning. Neural network models such as ResNet and VGG are used in this process. Through image analysis, the server identifies ingredients and dishes and extracts corresponding food component data. This data forms a crucial foundation for nutritional analysis based on user-submitted images.

[0430] Next, the server integrates the user's individual health data with food ingredient data. Users register data such as age, gender, and past health history from their device beforehand. Based on this, the server identifies the nutrients the user needs and generates personalized meal suggestions. For example, for a user who is deficient in vitamin C, the server can suggest recipes for an orange smoothie or a bell pepper salad.

[0431] The generated meal suggestions are sent from the server to the terminal, allowing the user to easily review them within the application. The user can then prepare their next meal based on the suggestions and ensure a balanced diet. An example of a prompt might be, "Please suggest a suitable vitamin D-rich dish for dinner." This allows users to efficiently manage their eating habits and receive support in maintaining their health.

[0432] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0433] Step 1:

[0434] The user takes a picture of their meal using a smartphone or tablet. They launch the camera app and take a picture that shows the entire meal. The input image data is saved in a ready-to-use format. This serves as the basic data for subsequent analysis processes.

[0435] Step 2:

[0436] The terminal sends captured image data to the server via a dedicated application. Here, the application is responsible for transferring the image files to the server in the correct format. The input is the image data sent from the terminal, and the output is the image data received by the server.

[0437] Step 3:

[0438] The server analyzes the received images using deep learning. Specifically, the server uses neural network models such as ResNet and VGG to identify ingredients and dishes within the images. The input to this process is image data, and the output is food component data. The analysis results in a list of ingredient names and dish names.

[0439] Step 4:

[0440] The server integrates the food component data obtained from image analysis with the user's individual health data. At this time, the server utilizes information previously provided by the user, such as age, gender, and health history. The input consists of food component data and individual health data, which are used to identify deficient nutrients. The output of this step is a list of deficient nutrients.

[0441] Step 5:

[0442] The server generates meal suggestions to supplement any nutritional deficiencies. This process uses a nutritional model to determine specific ingredients, cooking methods, and timing of intake. The meal suggestions generated by the server are customized for each user and designed to ensure efficient intake of specific nutrients. The input is a list of deficient nutrients, and the output is a personalized meal suggestion.

[0443] Step 6:

[0444] Meal suggestions generated by the server are sent to the terminal, and the user checks the suggestions through the application. The user uses this information to prepare their next meal and improve their nutritional balance. The input is the meal suggestions sent from the server, and the output is the user's plan for their next meal. Specifically, the user is notified when meal suggestions arrive via a notification function.

[0445] (Application Example 1)

[0446] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0447] Modern people lead busy lives, making it difficult to be mindful of nutritional balance in their daily meals. However, consuming a nutritionally balanced diet tailored to each individual is crucial for maintaining and improving health. Furthermore, choosing the optimal menu from many options is not easy. To solve these problems, there is a need for a means to accurately understand an individual's nutritional status and provide optimal nutrition without hassle.

[0448] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0449] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for selecting and providing a service to the user that proposes a nutritionally balanced menu to supplement the identified deficient nutrients. This makes it possible for the user to easily select and consume meals that take nutritional balance into consideration.

[0450] "User-submitted food images" refer to visual information of meals recorded by users using camera-equipped devices.

[0451] "Food composition data" refers to information such as nutrients and calories of each ingredient extracted from analyzed food images.

[0452] "Individual health data" refers to information about a specific user's health status, such as age, gender, and past health history.

[0453] "Nutrient deficiencies" refer to essential nutrients that users are not adequately obtaining through their current diet, based on their individual health data.

[0454] A "nutritionally balanced menu" is a meal plan that includes appropriate nutrients, suggested to supplement any nutritional deficiencies the user may have.

[0455] "The means of selecting and providing services to users" refers to a function that selects the most suitable service from a large number of options and guides the user to it, with the aim of supplementing the nutrients that the user is lacking.

[0456] To implement this invention, the server begins processing upon receiving food images captured by the user's terminal. The user takes photos of their meals with a mobile device such as a smartphone or tablet and sends them to the server through an application. The server is located in a cloud environment with high-speed processing capabilities and quickly analyzes the received image data. The analysis uses a deep learning algorithm that has been trained on a rich dataset beforehand, which allows for accurate identification of ingredients and dishes.

[0457] Next, the server uses the food component data obtained through analysis to compare it with the user's individual health data and identify any nutritional deficiencies. This individual health data includes personal elements such as the user's age, gender, and health history, and a nutritional model is built based on this data.

[0458] Once a nutritional deficiency is identified, the server selects the most suitable service to supplement it. Specifically, it accesses available delivery services and suggests nutritionally balanced menus. These suggestions are sent to the user's device, allowing them to easily place an order with a touch.

[0459] This system allows users to easily select nutritionally balanced meals in their daily lives. For example, if a vitamin D deficiency is detected from a breakfast image, a menu including "salmon and spinach salad" may be recommended for lunch. A possible example of a specific prompt message would be, "Analyze this image I took for breakfast and suggest a nutritionally balanced menu for lunch."

[0460] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0461] Step 1:

[0462] The device takes a picture of the meal and sends the image data to the server via the application. The input here is an image of the meal, and the output is the transmission of image data to the server.

[0463] Step 2:

[0464] The server analyzes received image data in the cloud using a deep learning algorithm. Based on the image data as input, food component data is extracted through analysis, and ingredients and dishes are identified. The output is food component data.

[0465] Step 3:

[0466] The server combines analyzed food component data with the user's individual health data to identify deficient nutrients. The inputs are food component data and individual health data, and the output is a list of deficient nutrients.

[0467] Step 4:

[0468] The server suggests nutritionally balanced menu options to supplement any missing nutrients. The server uses menu information from delivery services as input and generates an optimal meal plan tailored to the user's situation. The output is a list of optimal menu plans.

[0469] Step 5:

[0470] The terminal displays suggested menus received from the server to the user, offering delivery order options. The input is the suggested menu, and the output is the meal plan displayed to the user. The user can review this and place an order directly.

[0471] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0472] This invention combines a system for improving a user's nutritional status with an emotion engine that recognizes the user's emotions. The embodiments of this system are described in detail below.

[0473] First, the user takes a picture of their meal using a device such as a smartphone or tablet. The device then sends this image to a server via an application. The server uses a deep learning algorithm to analyze the received image and identify the ingredients and dishes. This process identifies the nutrients contained in the meal.

[0474] Next, the server receives emotional data acquired from the user's terminal through the emotion engine. This emotion engine analyzes data to identify the user's emotional state in their daily life. This includes functions to read emotions from the user's voice, facial expressions, and text input.

[0475] The server comprehensively analyzes nutritional data obtained from ingredients, individual user health data (age, gender, health history, etc.), and emotional data. This allows for meal suggestions that consider not only nutrition but also the user's emotional state. For example, if a user is feeling stressed, recipes using ingredients that are expected to have a relaxing effect will be suggested.

[0476] Meal suggestions are generated by the server and sent to the user's device. These suggestions include detailed recipes, a list of necessary ingredients, cooking instructions, and even tips for emotional well-being. Users can use this information to prepare meals, resulting in both physical nutritional balance and psychological satisfaction.

[0477] Thus, by adding an emotional element to nutritional management, the present invention realizes a user experience that was not possible with conventional meal management systems. Through this system, users can lead a healthier and more comfortable life.

[0478] The following describes the processing flow.

[0479] Step 1:

[0480] The user takes a picture of their meal using a smartphone or tablet. The device then imports the captured image data into the application in real time and prepares it.

[0481] Step 2:

[0482] The device sends the captured image to the server. This communication uses an encryption protocol to efficiently and securely transfer the image data.

[0483] Step 3:

[0484] The server analyzes the received images using deep learning algorithms to identify the ingredients and components of the food contained within the images. This reveals information about the nutrients contained in the meal.

[0485] Step 4:

[0486] The server also receives user emotion data collected from the terminal via the emotion engine. This data indicates the emotional state estimated from the user's tone of voice, facial expressions, input patterns, etc.

[0487] Step 5:

[0488] The server integrates and comprehensively analyzes nutritional data from ingredients, individual user health data (e.g., age, gender, health history), and emotional data. This analysis provides a comprehensive understanding of the user's health and emotional state, identifying deficient nutrients and emotional states that need improvement.

[0489] Step 6:

[0490] Based on the extracted information, the server generates meal suggestions that take into account the user's emotional state. These suggestions include ingredients and recipes that are expected to have a balanced nutritional profile and help stabilize emotions.

[0491] Step 7:

[0492] The generated meal suggestions are sent from the server to the terminal. Users can review these suggestions on their terminal and prepare their next meal using the suggested recipes and ingredients.

[0493] This series of steps allows users to easily select and implement meals that are tailored to their individual health and emotional state.

[0494] (Example 2)

[0495] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In modern society, individual users are required to maintain nutritional balance amidst diverse eating habits and living environments. However, conventional systems have struggled to provide meal suggestions that take into account the user's emotional state. This invention aims to solve the problem that conventional nutrition management systems cannot integrate the user's psychological state and nutritional status when making suggestions.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0498] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health information; and means for generating and presenting meal suggestions to supplement the identified deficient nutrients. This makes it possible to provide more appropriate nutritional suggestions that take into account the emotional state of each individual user.

[0499] "Food images" are image data taken by users for the purpose of recording their meals.

[0500] "Food composition data" refers to data that contains information about nutrients and ingredients identified through the analysis of food images.

[0501] "Individual health information" refers to information about each user's unique health status and history, including age, gender, and past health history.

[0502] "Nutrient deficiencies" refer to nutrients that are below the required amount, as determined based on the user's individual health information.

[0503] "Emotional data" refers to information that represents the user's psychological state, and includes data obtained from voice, facial expressions, text input, and other sources.

[0504] A "machine learning algorithm" is a computational method used in data analysis, aimed at identifying specific patterns or features.

[0505] This invention is a system that comprehensively analyzes a user's nutritional and emotional status and provides personalized meal suggestions. The main components of the system include image analysis, nutrient identification, emotional state analysis, and meal suggestion generation.

[0506] Users take pictures of their daily meals using devices such as smartphones and tablets. This image data is sent from the device to a server via the internet. The image data sent by the device is processed on the server.

[0507] The server analyzes the received images using machine learning algorithms, specifically deep learning technologies such as TensorFlow and PyTorch. This allows the ingredients and dishes identified from the images to be compared with a nutrient database, and their nutrients are identified.

[0508] In addition, users utilize the device's emotion engine to collect emotional data in their daily lives. This engine leverages speech analysis, image recognition, and text input analysis (e.g., using Google Cloud Speech-to-Text or Microsoft Azure Face API) to assess the user's psychological state. The assessed emotional data is then sent from the device to a server.

[0509] The server uses a generative AI model to perform a comprehensive analysis using nutrient data, emotional data, and the user's individual health information (age, gender, health history, etc.). The generative AI model uses the prompt message, "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress."

[0510] For example, if a user is feeling stressed, this system can suggest recipes using ingredients known to reduce stress. The meal suggestions generated by the server are sent to the user's device, where they receive detailed recipes, ingredient lists, and cooking instructions, allowing them to prepare meals that take their health and emotional state into consideration. Through this system, users can expect to achieve both physical health and psychological satisfaction simultaneously.

[0511] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0512] Step 1:

[0513] Users take pictures of their daily meals using their smartphones or tablets. These images serve as input data. The device automatically sends these images to the server. Specifically, the image data is compressed before transmission to improve communication efficiency.

[0514] Step 2:

[0515] The server analyzes image data received from the terminal using a deep learning algorithm. It converts the input image into features and performs calculations to identify specific ingredients and dishes. The analysis results output the identified ingredients and their nutritional information. This process utilizes a model based on TensorFlow.

[0516] Step 3:

[0517] The user's device routinely collects emotional data. This includes features that detect emotions from voice, facial expressions, and text input, which are treated as input data. The device uses an emotion engine to analyze the data and perform calculations to identify the user's psychological state. The analyzed emotional data is sent to a server as output. Google Cloud Speech-to-Text and Microsoft Azure Face APIs are utilized in this process.

[0518] Step 4:

[0519] The server integrates and analyzes nutrient data, the user's individual health information (including age, gender, and health history), and emotional data. This integrated data is used as input, and the prompt message "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress." is input to the generating AI model. This process performs calculations to generate appropriate meal suggestions based on the integrated data. The server then receives the resulting suggestions as output.

[0520] Step 5:

[0521] The server sends the generated meal suggestions to the user's device. The output meal suggestions include detailed recipes, a list of necessary ingredients, and cooking instructions. The user receives this information and uses it to plan their daily meals. Based on this information, the user can prepare meals that are optimal for their nutritional needs and emotional state.

[0522] (Application Example 2)

[0523] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] In modern society, maintaining individual health is becoming increasingly important, but conventional nutrition management systems have the challenge of not being able to suggest meals that take into account the user's emotional state. Furthermore, while it would be desirable to use autonomous robotic devices to efficiently manage individual meals at home, this is not currently possible with existing systems.

[0525] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0526] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; means for generating and presenting meal suggestions to supplement the identified deficient nutrients; means for analyzing the user's emotional state and customizing meal suggestions based on emotional data; and means for a robotic device to autonomously collect the user's emotions during meals and adjust the environment or provide advice based on the analysis results. This enables meal suggestions that comprehensively consider the user's health state and emotions, realizing effective meal management within the home.

[0527] "Food images" are image data taken by users to record the contents of their meals, and they serve as basic data for identifying food components.

[0528] "Food composition data" refers to information about nutrients and compounds contained in food ingredients and dishes, obtained by analyzing food images.

[0529] "Individual health data" refers to data that indicates the health status of an individual user, and includes information such as age, gender, and past health history.

[0530] "Emotional state" refers to the type and intensity of emotions a user experiences in their daily life or in specific situations, and is determined from factors such as voice and facial expressions.

[0531] A "robot device" is an autonomous electronic device intended to provide user support within the home, and has the function of collecting and analyzing emotional and environmental data to provide a user interface.

[0532] "Meal suggestions" refer to information about recommended meal menus and nutritional supplements that are generated taking into account the user's health condition and emotional state.

[0533] "Adjusting the environment" refers to the robotic device changing surrounding environmental elements in response to the user's emotional state, including, for example, the color of the lighting or the selection of music.

[0534] In this invention, a user takes a picture of their meal using a terminal and sends the image to a server. The server uses TensorFlow to analyze the received image and identify food component data. Because this analysis uses a deep learning algorithm, various ingredients and dishes can be accurately identified.

[0535] Next, the server references the user's individual health data to identify any nutritional deficiencies. This takes into account age, gender, and past health history. Furthermore, the user's device or robotic device uses an emotion engine to analyze the user's emotional state from their voice and facial expressions. This emotional data is analyzed using OpenCV for facial expression analysis and processed using natural language processing libraries for voice data.

[0536] The analyzed information is comprehensively analyzed by the server to generate meal suggestions that take into account the user's health and emotional state. These suggestions include recipes for dishes containing specific nutrients to stabilize emotions, as well as advice on adjusting the environment. For example, it might recommend herbal teas that are expected to have a relaxing effect, or playing calming music.

[0537] Furthermore, consumer robots autonomously provide assistance within the home in response to changes in the user's daily emotions and health. Through interaction with the user, robotic devices can provide an optimal environment. This system is expected to enable users to effectively manage both their physical health and psychological well-being.

[0538] For example, if a user is feeling stressed, the robot might suggest, "Why don't you try a relaxing herbal tea today? I'll dim the lights a little."

[0539] The prompt message to the generating AI model will be in the following format: "Suggest a relaxing meal menu for the user, especially if they are feeling stressed."

[0540] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0541] Step 1:

[0542] The user takes a picture of their meal using their device. This image is sent to the server. The input is the image data captured by the user, and the output is the image file to be sent to the server.

[0543] Step 2:

[0544] The server processes the received images and uses TensorFlow to leverage deep learning algorithms to analyze food components. The input is image data, and the output is identified food component data. Specifically, it converts the ingredients and dishes in the image into feature vectors and performs classification using a pre-trained model.

[0545] Step 3:

[0546] The server uses the analyzed food component data to compare it with the user's individual health data and identify any nutritional deficiencies. Input requires food component data and the user's health data, and output is a list of deficient nutrients. Data matching is performed using a database.

[0547] Step 4:

[0548] The user's device or robotic device analyzes emotional states from voice and facial expressions. Facial expression analysis is performed using OpenCV, and voice data is processed using a natural language processing library. Input is facial expression data (voice and images), and output is the user's emotion determination result.

[0549] Step 5:

[0550] The server comprehensively analyzes food component data, health data, nutrient deficiencies, and emotional data to generate meal suggestions tailored to the user. The input consists of various data, and the output is detailed information about the meal suggestions presented to the user. A generation AI model is used to provide menu suggestions that reflect individual factors to the greatest extent possible.

[0551] Step 6:

[0552] The robotic device communicates meal suggestions to the user and adjusts the environment as needed. Input is meal suggestion information from a server, and output is voice prompts and instructions for environmental adjustments. Specifically, it uses speech synthesis technology to communicate the suggestions verbally and changes environmental factors such as lighting color.

[0553] Step 7:

[0554] Users implement suggestions and enjoy their meals. This makes it possible to maintain physical health and psychological well-being.

[0555] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0556] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0558] [Fourth Embodiment]

[0559] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0560] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0561] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0563] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0565] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0566] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0567] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0568] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0569] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0570] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0571] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0572] This invention provides a system that enables users to consume nutritionally balanced meals on a daily basis. The following describes specific embodiments of this system.

[0573] First, users take pictures of their daily meals using a device such as a smartphone or tablet. The device then sends this image data to a server via an application. The server has high processing power in a cloud environment and can perform real-time image analysis.

[0574] The server uses a deep learning algorithm to analyze the received images and identify ingredients and dishes. This algorithm has been pre-trained on a rich dataset and is capable of pattern recognition of similar images.

[0575] Next, the server uses the analyzed food component data and the user's individual health data (including age, gender, and health history) to calculate which nutrients are deficient. This process is based on physiological and nutritional models.

[0576] Once the server identifies any nutrient deficiencies, it generates optimal meal suggestions to compensate for them. These suggestions include specific ingredients, cooking methods, and recommended meal timings. For example, if a vitamin C deficiency is detected, it might suggest recipes for orange smoothies or bell pepper salads.

[0577] The generated meal suggestions are sent from the server to the user's terminal, and the user can easily review the information. Based on these suggestions, the user can prepare their next meal and ensure nutritional balance.

[0578] This system allows users to improve their diet and maintain their health. The key feature of this invention is that it provides personalized suggestions tailored to the user's characteristics, achieving highly accurate nutritional management that cannot be obtained with conventional methods.

[0579] The following describes the processing flow.

[0580] Step 1:

[0581] The user takes a picture of their meal using a smartphone or tablet. The device then imports the resulting image directly into the application.

[0582] Step 2:

[0583] The device transmits the acquired meal image data to the server via the application. During this process, the image data is compressed or encrypted for secure and efficient transmission.

[0584] Step 3:

[0585] The server receives the transmitted image data and begins image analysis using a deep learning algorithm. It utilizes a pre-trained neural network model to accurately recognize ingredients and dishes within the image.

[0586] Step 4:

[0587] The server retrieves relevant food component data from a database based on the food ingredient information obtained through image analysis. This creates a detailed profile of the nutrients contained in the image.

[0588] Step 5:

[0589] The server integrates the user's individual health data (age, gender, health history, etc.) with food composition data to perform analysis to identify nutritional deficiencies. It then calculates and evaluates the user's current dietary status based on physiological models and nutritional theories.

[0590] Step 6:

[0591] The server generates meal suggestions to address identified nutrient deficiencies. These suggestions include recipes and cooking instructions using nutrient-rich ingredients. The suggestions are customized to take into account the user's preferences and allergy information.

[0592] Step 7:

[0593] The server sends the generated meal suggestions to the user's device. The user can review the suggestions displayed on their device and use them to prepare their next meal. This information supports the user in making decisions to maintain a healthy diet.

[0594] (Example 1)

[0595] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0596] In today's busy lifestyle, it is difficult for users to easily manage the nutritional balance of their daily meals. In particular, making effective meal choices that take into account individual health conditions and nutrient deficiencies is difficult without specialized knowledge. There is a need to solve this problem.

[0597] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0598] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for integrating the food component data and individual health data to generate and present meal suggestions to improve nutritional balance. This makes it possible for the user to easily select a nutritionally balanced meal that suits them and maintain their health.

[0599] A "user" refers to an individual who takes pictures of food and receives personalized nutritional management services based on that data.

[0600] "Food images" refer to image data of meals or ingredients taken by users.

[0601] "Food composition data" refers to information about the nutrients and components contained in food ingredients and dishes, obtained by analyzing food images.

[0602] "Individual health data" refers to personal health information related to nutrition management, such as the user's age, gender, and past medical history.

[0603] "Nutrients" refer to important substances found in food and dishes that contribute to maintaining human health and improving quality of life.

[0604] "Meal suggestions" refer to information that presents suitable ingredients and cooking options to the user, based on food composition data and individual health data.

[0605] A "server" refers to a device located in a cloud environment that possesses the computing resources and functions necessary for analyzing food images and managing nutritional information.

[0606] A "machine learning algorithm" refers to a computational method that allows computers to automatically learn patterns from data and analyze unknown data.

[0607] This system combines multiple technological elements to help users consume nutritionally balanced meals on a daily basis. Users take pictures of each meal using a smartphone or tablet. This allows users to easily collect and manage their meal data.

[0608] The device transmits captured image data to a server via a dedicated application. Typical hardware such as smartphones and tablets are expected to be used. The server possesses high computing power in a cloud environment and the ability to process image data in real time. Specifically, a commercial cloud service platform can be utilized.

[0609] The server analyzes the received images using machine learning algorithms based on deep learning. Neural network models such as ResNet and VGG are used in this process. Through image analysis, the server identifies ingredients and dishes and extracts corresponding food component data. This data forms a crucial foundation for nutritional analysis based on user-submitted images.

[0610] Next, the server integrates the user's individual health data with food ingredient data. Users register data such as age, gender, and past health history from their device beforehand. Based on this, the server identifies the nutrients the user needs and generates personalized meal suggestions. For example, for a user who is deficient in vitamin C, the server can suggest recipes for an orange smoothie or a bell pepper salad.

[0611] The generated meal suggestions are sent from the server to the terminal, allowing the user to easily review them within the application. The user can then prepare their next meal based on the suggestions and ensure a balanced diet. An example of a prompt might be, "Please suggest a suitable vitamin D-rich dish for dinner." This allows users to efficiently manage their eating habits and receive support in maintaining their health.

[0612] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0613] Step 1:

[0614] The user takes a picture of their meal using a smartphone or tablet. They launch the camera app and take a picture that shows the entire meal. The input image data is saved in a ready-to-use format. This serves as the basic data for subsequent analysis processes.

[0615] Step 2:

[0616] The terminal sends captured image data to the server via a dedicated application. Here, the application is responsible for transferring the image files to the server in the correct format. The input is the image data sent from the terminal, and the output is the image data received by the server.

[0617] Step 3:

[0618] The server analyzes the received images using deep learning. Specifically, the server uses neural network models such as ResNet and VGG to identify ingredients and dishes within the images. The input to this process is image data, and the output is food component data. The analysis results in a list of ingredient names and dish names.

[0619] Step 4:

[0620] The server integrates the food component data obtained from image analysis with the user's individual health data. At this time, the server utilizes information previously provided by the user, such as age, gender, and health history. The input consists of food component data and individual health data, which are used to identify deficient nutrients. The output of this step is a list of deficient nutrients.

[0621] Step 5:

[0622] The server generates meal suggestions to supplement any nutritional deficiencies. This process uses a nutritional model to determine specific ingredients, cooking methods, and timing of intake. The meal suggestions generated by the server are customized for each user and designed to ensure efficient intake of specific nutrients. The input is a list of deficient nutrients, and the output is a personalized meal suggestion.

[0623] Step 6:

[0624] Meal suggestions generated by the server are sent to the terminal, and the user checks the suggestions through the application. The user uses this information to prepare their next meal and improve their nutritional balance. The input is the meal suggestions sent from the server, and the output is the user's plan for their next meal. Specifically, the user is notified when meal suggestions arrive via a notification function.

[0625] (Application Example 1)

[0626] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0627] Modern people lead busy lives, making it difficult to be mindful of nutritional balance in their daily meals. However, consuming a nutritionally balanced diet tailored to each individual is crucial for maintaining and improving health. Furthermore, choosing the optimal menu from many options is not easy. To solve these problems, there is a need for a means to accurately understand an individual's nutritional status and provide optimal nutrition without hassle.

[0628] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0629] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; and means for selecting and providing a service to the user that proposes a nutritionally balanced menu to supplement the identified deficient nutrients. This makes it possible for the user to easily select and consume meals that take nutritional balance into consideration.

[0630] "User-submitted food images" refer to visual information of meals recorded by users using camera-equipped devices.

[0631] "Food composition data" refers to information such as nutrients and calories of each ingredient extracted from analyzed food images.

[0632] "Individual health data" refers to information about a specific user's health status, such as age, gender, and past health history.

[0633] "Nutrient deficiencies" refer to essential nutrients that users are not adequately obtaining through their current diet, based on their individual health data.

[0634] A "nutritionally balanced menu" is a meal plan that includes appropriate nutrients, suggested to supplement any nutritional deficiencies the user may have.

[0635] "The means of selecting and providing services to users" refers to a function that selects the most suitable service from a large number of options and guides the user to it, with the aim of supplementing the nutrients that the user is lacking.

[0636] To implement this invention, the server begins processing upon receiving food images captured by the user's terminal. The user takes photos of their meals with a mobile device such as a smartphone or tablet and sends them to the server through an application. The server is located in a cloud environment with high-speed processing capabilities and quickly analyzes the received image data. The analysis uses a deep learning algorithm that has been trained on a rich dataset beforehand, which allows for accurate identification of ingredients and dishes.

[0637] Next, the server uses the food component data obtained through analysis to compare it with the user's individual health data and identify any nutritional deficiencies. This individual health data includes personal elements such as the user's age, gender, and health history, and a nutritional model is built based on this data.

[0638] Once a nutritional deficiency is identified, the server selects the most suitable service to supplement it. Specifically, it accesses available delivery services and suggests nutritionally balanced menus. These suggestions are sent to the user's device, allowing them to easily place an order with a touch.

[0639] This system allows users to easily select nutritionally balanced meals in their daily lives. For example, if a vitamin D deficiency is detected from a breakfast image, a menu including "salmon and spinach salad" may be recommended for lunch. A possible example of a specific prompt message would be, "Analyze this image I took for breakfast and suggest a nutritionally balanced menu for lunch."

[0640] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0641] Step 1:

[0642] The device takes a picture of the meal and sends the image data to the server via the application. The input here is an image of the meal, and the output is the transmission of image data to the server.

[0643] Step 2:

[0644] The server analyzes received image data in the cloud using a deep learning algorithm. Based on the image data as input, food component data is extracted through analysis, and ingredients and dishes are identified. The output is food component data.

[0645] Step 3:

[0646] The server combines analyzed food component data with the user's individual health data to identify deficient nutrients. The inputs are food component data and individual health data, and the output is a list of deficient nutrients.

[0647] Step 4:

[0648] The server suggests nutritionally balanced menu options to supplement any missing nutrients. The server uses menu information from delivery services as input and generates an optimal meal plan tailored to the user's situation. The output is a list of optimal menu plans.

[0649] Step 5:

[0650] The terminal displays suggested menus received from the server to the user, offering delivery order options. The input is the suggested menu, and the output is the meal plan displayed to the user. The user can review this and place an order directly.

[0651] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0652] This invention combines a system for improving a user's nutritional status with an emotion engine that recognizes the user's emotions. The embodiments of this system are described in detail below.

[0653] First, the user takes a picture of their meal using a device such as a smartphone or tablet. The device then sends this image to a server via an application. The server uses a deep learning algorithm to analyze the received image and identify the ingredients and dishes. This process identifies the nutrients contained in the meal.

[0654] Next, the server receives emotional data acquired from the user's terminal through the emotion engine. This emotion engine analyzes data to identify the user's emotional state in their daily life. This includes functions to read emotions from the user's voice, facial expressions, and text input.

[0655] The server comprehensively analyzes nutritional data obtained from ingredients, individual user health data (age, gender, health history, etc.), and emotional data. This allows for meal suggestions that consider not only nutrition but also the user's emotional state. For example, if a user is feeling stressed, recipes using ingredients that are expected to have a relaxing effect will be suggested.

[0656] Meal suggestions are generated by the server and sent to the user's device. These suggestions include detailed recipes, a list of necessary ingredients, cooking instructions, and even tips for emotional well-being. Users can use this information to prepare meals, resulting in both physical nutritional balance and psychological satisfaction.

[0657] Thus, by adding an emotional element to nutritional management, the present invention realizes a user experience that was not possible with conventional meal management systems. Through this system, users can lead a healthier and more comfortable life.

[0658] The following describes the processing flow.

[0659] Step 1:

[0660] The user takes a picture of their meal using a smartphone or tablet. The device then imports the captured image data into the application in real time and prepares it.

[0661] Step 2:

[0662] The device sends the captured image to the server. This communication uses an encryption protocol to efficiently and securely transfer the image data.

[0663] Step 3:

[0664] The server analyzes the received images using deep learning algorithms to identify the ingredients and components of the food contained within the images. This reveals information about the nutrients contained in the meal.

[0665] Step 4:

[0666] The server also receives user emotion data collected from the terminal via the emotion engine. This data indicates the emotional state estimated from the user's tone of voice, facial expressions, input patterns, etc.

[0667] Step 5:

[0668] The server integrates and comprehensively analyzes nutritional data from ingredients, individual user health data (e.g., age, gender, health history), and emotional data. This analysis provides a comprehensive understanding of the user's health and emotional state, identifying deficient nutrients and emotional states that need improvement.

[0669] Step 6:

[0670] Based on the extracted information, the server generates meal suggestions that take into account the user's emotional state. These suggestions include ingredients and recipes that are expected to have a balanced nutritional profile and help stabilize emotions.

[0671] Step 7:

[0672] The generated meal suggestions are sent from the server to the terminal. Users can review these suggestions on their terminal and prepare their next meal using the suggested recipes and ingredients.

[0673] This series of steps allows users to easily select and implement meals that are tailored to their individual health and emotional state.

[0674] (Example 2)

[0675] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] In modern society, individual users are required to maintain nutritional balance amidst diverse eating habits and living environments. However, conventional systems have struggled to provide meal suggestions that take into account the user's emotional state. This invention aims to solve the problem that conventional nutrition management systems cannot integrate the user's psychological state and nutritional status when making suggestions.

[0677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0678] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health information; and means for generating and presenting meal suggestions to supplement the identified deficient nutrients. This makes it possible to provide more appropriate nutritional suggestions that take into account the emotional state of each individual user.

[0679] "Food images" are image data taken by users for the purpose of recording their meals.

[0680] "Food composition data" refers to data that contains information about nutrients and ingredients identified through the analysis of food images.

[0681] "Individual health information" refers to information about each user's unique health status and history, including age, gender, and past health history.

[0682] "Nutrient deficiencies" refer to nutrients that are below the required amount, as determined based on the user's individual health information.

[0683] "Emotional data" refers to information that represents the user's psychological state, and includes data obtained from voice, facial expressions, text input, and other sources.

[0684] A "machine learning algorithm" is a computational method used in data analysis, aimed at identifying specific patterns or features.

[0685] This invention is a system that comprehensively analyzes a user's nutritional and emotional status and provides personalized meal suggestions. The main components of the system include image analysis, nutrient identification, emotional state analysis, and meal suggestion generation.

[0686] Users take pictures of their daily meals using devices such as smartphones and tablets. This image data is sent from the device to a server via the internet. The image data sent by the device is processed on the server.

[0687] The server analyzes the received images using machine learning algorithms, specifically deep learning technologies such as TensorFlow and PyTorch. This allows the ingredients and dishes identified from the images to be compared with a nutrient database, and their nutrients are identified.

[0688] In addition, users utilize the device's emotion engine to collect emotional data in their daily lives. This engine leverages speech analysis, image recognition, and text input analysis (e.g., using Google Cloud Speech-to-Text or Microsoft Azure Face API) to assess the user's psychological state. The assessed emotional data is then sent from the device to a server.

[0689] The server uses a generative AI model to perform a comprehensive analysis using nutrient data, emotional data, and the user's individual health information (age, gender, health history, etc.). The generative AI model uses the prompt message, "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress."

[0690] For example, if a user is feeling stressed, this system can suggest recipes using ingredients known to reduce stress. The meal suggestions generated by the server are sent to the user's device, where they receive detailed recipes, ingredient lists, and cooking instructions, allowing them to prepare meals that take their health and emotional state into consideration. Through this system, users can expect to achieve both physical health and psychological satisfaction simultaneously.

[0691] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0692] Step 1:

[0693] Users take pictures of their daily meals using their smartphones or tablets. These images serve as input data. The device automatically sends these images to the server. Specifically, the image data is compressed before transmission to improve communication efficiency.

[0694] Step 2:

[0695] The server analyzes image data received from the terminal using a deep learning algorithm. It converts the input image into features and performs calculations to identify specific ingredients and dishes. The analysis results output the identified ingredients and their nutritional information. This process utilizes a model based on TensorFlow.

[0696] Step 3:

[0697] The user's device routinely collects emotional data. This includes features that detect emotions from voice, facial expressions, and text input, which are treated as input data. The device uses an emotion engine to analyze the data and perform calculations to identify the user's psychological state. The analyzed emotional data is sent to a server as output. Google Cloud Speech-to-Text and Microsoft Azure Face APIs are utilized in this process.

[0698] Step 4:

[0699] The server integrates and analyzes nutrient data, the user's individual health information (including age, gender, and health history), and emotional data. This integrated data is used as input, and the prompt message "Based on the user's emotional state and health data, please suggest a meal recipe that will help reduce stress." is input to the generating AI model. This process performs calculations to generate appropriate meal suggestions based on the integrated data. The server then receives the resulting suggestions as output.

[0700] Step 5:

[0701] The server sends the generated meal suggestions to the user's device. The output meal suggestions include detailed recipes, a list of necessary ingredients, and cooking instructions. The user receives this information and uses it to plan their daily meals. Based on this information, the user can prepare meals that are optimal for their nutritional needs and emotional state.

[0702] (Application Example 2)

[0703] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] In modern society, maintaining individual health is becoming increasingly important, but conventional nutrition management systems have the challenge of not being able to suggest meals that take into account the user's emotional state. Furthermore, while it would be desirable to use autonomous robotic devices to efficiently manage individual meals at home, this is not currently possible with existing systems.

[0705] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0706] In this invention, the server includes means for receiving food images taken by the user and analyzing the images to identify food component data; means for identifying deficient nutrients based on the analyzed food component data and the user's individual health data; means for generating and presenting meal suggestions to supplement the identified deficient nutrients; means for analyzing the user's emotional state and customizing meal suggestions based on emotional data; and means for a robotic device to autonomously collect the user's emotions during meals and adjust the environment or provide advice based on the analysis results. This enables meal suggestions that comprehensively consider the user's health state and emotions, realizing effective meal management within the home.

[0707] "Food images" are image data taken by users to record the contents of their meals, and they serve as basic data for identifying food components.

[0708] "Food composition data" refers to information about nutrients and compounds contained in food ingredients and dishes, obtained by analyzing food images.

[0709] "Individual health data" refers to data that indicates the health status of an individual user, and includes information such as age, gender, and past health history.

[0710] "Emotional state" refers to the type and intensity of emotions a user experiences in their daily life or in specific situations, and is determined from factors such as voice and facial expressions.

[0711] A "robot device" is an autonomous electronic device intended to provide user support within the home, and has the function of collecting and analyzing emotional and environmental data to provide a user interface.

[0712] "Meal suggestions" refer to information about recommended meal menus and nutritional supplements that are generated taking into account the user's health condition and emotional state.

[0713] "Adjusting the environment" refers to the robotic device changing surrounding environmental elements in response to the user's emotional state, including, for example, the color of the lighting or the selection of music.

[0714] In this invention, a user takes a picture of their meal using a terminal and sends the image to a server. The server uses TensorFlow to analyze the received image and identify food component data. Because this analysis uses a deep learning algorithm, various ingredients and dishes can be accurately identified.

[0715] Next, the server references the user's individual health data to identify any nutritional deficiencies. This takes into account age, gender, and past health history. Furthermore, the user's device or robotic device uses an emotion engine to analyze the user's emotional state from their voice and facial expressions. This emotional data is analyzed using OpenCV for facial expression analysis and processed using natural language processing libraries for voice data.

[0716] The analyzed information is comprehensively analyzed by the server to generate meal suggestions that take into account the user's health and emotional state. These suggestions include recipes for dishes containing specific nutrients to stabilize emotions, as well as advice on adjusting the environment. For example, it might recommend herbal teas that are expected to have a relaxing effect, or playing calming music.

[0717] Furthermore, consumer robots autonomously provide assistance within the home in response to changes in the user's daily emotions and health. Through interaction with the user, robotic devices can provide an optimal environment. This system is expected to enable users to effectively manage both their physical health and psychological well-being.

[0718] For example, if a user is feeling stressed, the robot might suggest, "Why don't you try a relaxing herbal tea today? I'll dim the lights a little."

[0719] The prompt message to the generating AI model will be in the following format: "Suggest a relaxing meal menu for the user, especially if they are feeling stressed."

[0720] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0721] Step 1:

[0722] The user takes a picture of their meal using their device. This image is sent to the server. The input is the image data captured by the user, and the output is the image file to be sent to the server.

[0723] Step 2:

[0724] The server processes the received images and uses TensorFlow to leverage deep learning algorithms to analyze food components. The input is image data, and the output is identified food component data. Specifically, it converts the ingredients and dishes in the image into feature vectors and performs classification using a pre-trained model.

[0725] Step 3:

[0726] The server uses the analyzed food component data to compare it with the user's individual health data and identify any nutritional deficiencies. Input requires food component data and the user's health data, and output is a list of deficient nutrients. Data matching is performed using a database.

[0727] Step 4:

[0728] The user's device or robotic device analyzes emotional states from voice and facial expressions. Facial expression analysis is performed using OpenCV, and voice data is processed using a natural language processing library. Input is facial expression data (voice and images), and output is the user's emotion determination result.

[0729] Step 5:

[0730] The server comprehensively analyzes food component data, health data, nutrient deficiencies, and emotional data to generate meal suggestions tailored to the user. The input consists of various data, and the output is detailed information about the meal suggestions presented to the user. A generation AI model is used to provide menu suggestions that reflect individual factors to the greatest extent possible.

[0731] Step 6:

[0732] The robotic device communicates meal suggestions to the user and adjusts the environment as needed. Input is meal suggestion information from a server, and output is voice prompts and instructions for environmental adjustments. Specifically, it uses speech synthesis technology to communicate the suggestions verbally and changes environmental factors such as lighting color.

[0733] Step 7:

[0734] Users implement suggestions and enjoy their meals. This makes it possible to maintain physical health and psychological well-being.

[0735] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0736] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0737] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0738] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0739] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0740] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0741] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0742] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0743] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0744] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0745] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0746] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0747] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0748] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0749] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0750] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0751] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0752] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0753] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0754] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0755] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0756] The following is further disclosed regarding the embodiments described above.

[0757] (Claim 1)

[0758] A means for receiving food images taken by a user, analyzing the images, and identifying food component data,

[0759] Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data,

[0760] A means of generating and presenting dietary suggestions to the user to supplement identified nutrient deficiencies,

[0761] A system that includes this.

[0762] (Claim 2)

[0763] The system according to claim 1, comprising means for identifying food components using a deep learning algorithm for analyzing food images.

[0764] (Claim 3)

[0765] The system according to claim 1, comprising means for considering age, gender, and past health history as individual health data of the user.

[0766] "Example 1"

[0767] (Claim 1)

[0768] A means for receiving food images taken by a user, analyzing the images, and identifying food component data,

[0769] Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data,

[0770] A means of integrating food component data and individual health data to generate and present meal suggestions to improve nutritional balance to the user,

[0771] ...

[0772] A system that includes this.

[0773] (Claim 2)

[0774] The system according to claim 1, comprising means for identifying food components using a machine learning algorithm for analyzing food images.

[0775] (Claim 3)

[0776] The system according to claim 1, comprising means for considering age, gender, and past medical history as individual health data of the user.

[0777] "Application Example 1"

[0778] (Claim 1)

[0779] A means for receiving food images taken by a user, analyzing the images, and identifying food component data,

[0780] Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data,

[0781] We select a service that proposes nutritionally balanced menus to supplement identified nutrient deficiencies, and provide this service to the user.

[0782] A system that includes this.

[0783] (Claim 2)

[0784] The system according to claim 1, comprising means for identifying food components using a deep learning algorithm for analyzing food images.

[0785] (Claim 3)

[0786] The system according to claim 1, comprising means for considering age, gender, and past health history as individual health data of the user.

[0787] "Example 2 of combining an emotion engine"

[0788] (Claim 1)

[0789] A means for receiving food images taken by a user, analyzing the images, and identifying food component data,

[0790] Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health information,

[0791] A means of generating and presenting dietary suggestions to the user to supplement identified nutrient deficiencies,

[0792] A means of collecting and analyzing emotional data to identify the emotional state of users,

[0793] A means of adjusting meal suggestions considering analyzed emotional data,

[0794] A system that includes this.

[0795] (Claim 2)

[0796] The system according to claim 1, comprising means for identifying food components using a machine learning algorithm for analyzing food images.

[0797] (Claim 3)

[0798] The system according to claim 1, comprising means for considering age, gender, and health history as individual health information of the user.

[0799] "Application example 2 when combining with an emotional engine"

[0800] (Claim 1)

[0801] A means for receiving food images taken by a user, analyzing the images, and identifying food component data,

[0802] Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data,

[0803] A means of generating and presenting dietary suggestions to the user to supplement identified nutrient deficiencies,

[0804] A means of analyzing the user's emotional state and customizing meal suggestions based on emotional data,

[0805] A means by which a robotic device autonomously collects user emotions during meals and adjusts the environment or provides advice based on the analysis results,

[0806] A system that includes this.

[0807] (Claim 2)

[0808] The system according to claim 1, comprising means for identifying food components using a deep learning algorithm for analyzing food images.

[0809] (Claim 3)

[0810] The system according to claim 1, comprising means for considering age, gender, and past health history as individual health data of the user, and further comprising means for determining emotions through the user's voice and facial expressions. [Explanation of symbols]

[0811] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving food images taken by a user, analyzing the images, and identifying food component data, Based on the analyzed food component data, a means of identifying deficient nutrients based on the user's individual health data, We select a service that proposes nutritionally balanced menus to supplement identified nutrient deficiencies, and provide this service to the user. A system that includes this.

2. The system according to claim 1, comprising means for identifying food components using a deep learning algorithm for analyzing food images.

3. The system according to claim 1, comprising means for considering age, gender, and past health history as individual health data of the user.

Images