system

Abstract

The system according to this embodiment aims to allow users to easily manage their meals. [Solution] A system according to an embodiment includes a photographing unit, an analysis unit, a provision unit, and a history management unit. The photographing unit allows a user to take a photo of a meal. The analysis unit analyzes the photo taken by the photographing unit, recognizes ingredients, and calculates nutrients. The provision unit provides nutrient balance, calories, and dietary improvements or advice based on the analysis results obtained by the analysis unit. The history management unit manages the user's diet history based on the information provided by the provision unit, compares it with past data, and provides advice.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have had the problem that dietary management is time-consuming and difficult to continue.

[0005] The system according to this embodiment aims to allow users to easily manage their meals. [Means for solving the problem]

[0006] The system according to this embodiment comprises a shooting unit, an analysis unit, a serving unit, and a history management unit. The shooting unit takes a photograph of the user's meal. The analysis unit analyzes the photograph taken by the shooting unit, recognizes the ingredients, and calculates the nutrients. Based on the analysis results obtained by the analysis unit, the serving unit provides information on nutrient balance, calories, and suggestions for improving the meal or advice. The history management unit manages the user's meal history based on the information provided by the serving unit and provides advice by comparing it with past data. [Effects of the Invention]

[0007] The system according to this embodiment allows users to easily manage their meals. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A health management system according to an embodiment of the present invention analyzes a user's meal content and provides advice and advice on nutrient balance and calories, as well as suggestions for improving the diet, simply by the user taking a photo of the meal. This health management system allows the user to take a photo of the meal, analyze the photo, recognize ingredients, and calculate nutrients. Furthermore, based on the analysis results, the system provides advice and advice on nutrient balance and calories, as well as suggestions for improving the diet. For example, a user simply takes a photo of their breakfast, lunch, and dinner, and the system automatically analyzes the meal content, calculates the nutrient balance and calories, and provides these to the user. The system also manages the user's diet history and compares it with past data to provide advice. For example, if there is an imbalance in nutrients based on the diet data from the past week, the system provides advice on how to improve the diet. This allows the user to easily manage their diet. Thus, the health management system allows the user to analyze a meal content and provide advice and advice on nutrient balance and calories, as well as suggestions for improving the diet, simply by the user taking a photo of the meal.

[0029] A health management system according to an embodiment includes a photographing unit, an analysis unit, a providing unit, and a history management unit. The photographing unit allows a user to take photos of their meals. When taking photos of meals, the user simply takes photos of the meals with a camera such as a smartphone without any special operations. For example, the user takes photos of breakfast, lunch, and dinner. This information is input into the system. The analysis unit analyzes the photos taken by the photographing unit, recognizes ingredients, and calculates the nutritional value of each. The analysis unit, for example, uses AI to recognize ingredients in the photos and calculates the nutritional value of each. For example, the analysis unit identifies ingredients such as vegetables, meat, and fish in the photos and calculates the calories and nutrients of each. The providing unit provides the user with information on nutrient balance and calories, as well as advice and suggestions for improving their diet, based on the analysis results obtained by the analysis unit. For example, the providing unit provides the user with information on nutrient balance and calories, as well as advice and suggestions for improving their diet. For example, if the user's diet is unbalanced based on photos taken by the user, the providing unit provides advice on how to improve the diet. The history management unit manages the user's diet history based on the information provided by the providing unit and compares it with past data to provide advice. The history management unit, for example, manages the user's diet history and compares it with past data to provide advice. For example, if the user's nutritional balance is unbalanced based on the dietary data from the past week, advice on how to improve the balance can be provided. This allows the health management system according to the embodiment to analyze the contents of a meal and provide advice on the balance of nutrients, calories, and how to improve the diet simply by the user taking a photo of the meal.

[0030] The analysis unit can recognize ingredients in a photo and calculate the nutritional value of each ingredient. The analysis unit can use, for example, AI to recognize ingredients in a photo and calculate the nutritional value of each ingredient. For example, it can identify ingredients such as vegetables, meat, and fish in the photo and calculate the calories and nutrients of each ingredient. The analysis unit can also use an image recognition algorithm to recognize ingredients and calculate nutrients by referencing a database. For example, the image recognition algorithm can identify ingredients in the photo and obtain nutrient information for those ingredients from a database for calculation. This allows accurate nutritional information to be provided by recognizing ingredients in the photo and calculating their nutritional value. Some or all of the above-described processing in the analysis unit can be performed using, for example, AI, or without AI. For example, the analysis unit can input the recognition of ingredients in the photo and the calculation of nutrients into an AI model and use the analysis results output by the AI ​​model.

[0031] The providing unit can provide nutrient balance, calories, and dietary improvements or advice. The providing unit provides the user with nutrient balance, calories, dietary improvements, and advice based on, for example, the analysis results obtained by the analysis unit. For example, if the nutrient balance is unbalanced based on a photo taken by the user, the providing unit can provide advice on how to improve the nutrient balance. The providing unit can also calculate nutrient balance and calories and provide the user with the same. For example, based on the nutrient balance and calories calculated by the analysis unit, the providing unit can provide the user with dietary improvements or advice. This can help the user improve their eating habits by providing nutrient balance, calories, dietary improvements, and advice. Some or all of the above-described processing by the providing unit can be performed using, for example, AI, or can be performed without using AI. For example, the providing unit can input the analysis results obtained by the analysis unit into an AI model and use the advice output by the AI ​​model.

[0032] The history management unit can manage the user's diet history and provide advice by comparing it with past data. For example, the history management unit can manage the user's diet history and provide advice by comparing it with past data. For example, if there is an imbalance in nutrients based on diet data from the past week, the history management unit can provide advice on how to improve the imbalance. The history management unit can also store the user's diet history in a database and provide advice by comparing it with past data. For example, the history management unit can store the user's diet history in a database and provide advice by comparing it with past data to analyze nutrient balance and calorie fluctuations. This allows the user to continuously improve their diet by managing their diet history and providing advice by comparing it with past data. Some or all of the above-described processing in the history management unit can be performed using, for example, AI, or without AI. For example, the history management unit can input the user's diet history into an AI model and use the advice output by the AI ​​model.

[0033] The photographing unit can suggest the optimal photographing method by referring to the user's past meal history when photographing. For example, the photographing unit can suggest the optimal photographing method by referring to the user's past meal history when photographing. For example, the photographing unit can analyze photos of meals taken by the user in the past and suggest the most effective photographing angle. The photographing unit can also adjust the photographing method based on the user's past meal history so that specific ingredients are best captured. For example, the photographing unit can adjust the photographing method based on the user's past meal history so that specific ingredients are best captured. The photographing unit can also optimize the lighting conditions during photographing based on the user's past meal history. For example, the lighting conditions during photographing are optimized based on the user's past meal history. This allows for more effective photographs of food to be taken by suggesting the optimal photographing method by referring to the user's past meal history. Some or all of the above-described processing in the photographing unit may be performed using, for example, AI, or may be performed without AI. For example, the photographing unit can input the user's past meal history data into a generation AI and use the optimal photographing method output by the generation AI.

[0034] The camera unit can automatically switch the shooting mode depending on the type and amount of food when taking a photo. The camera unit, for example, automatically switches the shooting mode depending on the type and amount of food when taking a photo. For example, when a large dish is being served, the camera unit switches to wide-angle mode to capture the entire dish. In addition, when a small dish is being served, the camera unit can switch to macro mode to capture the details. For example, when a small dish is being served, the camera unit switches to macro mode to capture the details. In addition, when a drink is included, the camera unit can apply a specific filter to emphasize the color. For example, when a drink is included, the camera unit applies a specific filter to emphasize the color. In this way, the optimal photo can be taken by automatically switching the shooting mode depending on the type and amount of food. Some or all of the above-mentioned processing in the camera unit may be performed using, or without, AI. For example, the camera unit can input data on the type and amount of food into the generation AI and use the shooting mode output by the generation AI.

[0035] The photographing unit can prioritize photographing highly relevant meals in consideration of the user's geographical location information when photographing. For example, the photographing unit prioritizes photographing highly relevant meals in consideration of the user's geographical location information when photographing. For example, when the user is traveling, the photographing unit prioritizes photographing local specialties. Furthermore, when the user is at a restaurant, the photographing unit can prioritize photographing popular dishes on the menu. For example, when the user is at a restaurant, the photographing unit prioritizes photographing popular dishes on the menu. Furthermore, when the user is at home, the photographing unit can prioritize photographing dishes that the user eats on a daily basis. For example, when the user is at home, the photographing unit prioritizes photographing dishes that the user eats on a daily basis. In this way, by prioritizing photographing highly relevant meals in consideration of the user's geographical location information, it is possible to collect region-specific meal information. Some or all of the above-described processing by the photographing unit may be performed using, for example, AI, or may be performed without using AI. For example, the photographing unit can input the user's geographical location information data into a generation AI and use the highly relevant meals output by the generation AI.

[0036] The photography unit can analyze the user's social media activity during photography and photograph related meals. For example, the photography unit can prioritize photographing dishes that the user has shared on social media. The photography unit can also prioritize photographing dishes that have been featured by influencers that the user follows. For example, the photography unit can prioritize photographing dishes that have been featured by influencers that the user follows. The photography unit can also prioritize photographing dishes that the user has "liked". For example, the photography unit can prioritize photographing dishes that have been "liked". By analyzing the user's social media activity and photographing related meals, it is possible to collect food information that matches the user's interests. Some or all of the above processing in the photography unit may be performed using AI, for example, or without AI. For example, the photography unit can input the user's social media activity data into a generating AI and use the related meals output by the generating AI.

[0037] The analysis unit can calculate nutrients while considering the freshness and cooking method of the ingredients during analysis. For example, it can calculate nutrients while considering the difference in nutrients between fresh vegetables and frozen vegetables. The analysis unit can also calculate nutrients while considering the difference in nutrients between grilled fish and raw fish. For example, it can calculate nutrients while considering the difference in nutrients between grilled fish and raw fish. The analysis unit can also calculate nutrients while considering the difference in nutrients between dishes with long cooking times and dishes with short cooking times. For example, it can calculate nutrients while considering the difference in nutrients between dishes with long cooking times and dishes with short cooking times. By calculating nutrients while considering the freshness and cooking method of the ingredients, more accurate nutritional information can be provided. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input data on the freshness and cooking method of the ingredients into a generating AI and use the nutrient calculation results output by the generating AI.

[0038] The analysis unit can calculate nutrients while considering the origin and production methods of the ingredients during analysis. For example, it can calculate nutrients while considering the differences in nutrients between organically grown vegetables and conventionally grown vegetables. The analysis unit can also calculate nutrients while considering the differences in nutrients between locally produced and imported ingredients. For example, it can calculate nutrients while considering the differences in nutrients between locally produced and imported ingredients. The analysis unit can also calculate nutrients while considering the differences in nutrients between meat from free-range cattle and meat from factory-produced cattle. For example, it can calculate nutrients while considering the differences in nutrients between meat from free-range cattle and meat from factory-produced cattle. By calculating nutrients while considering the origin and production methods of the ingredients, more accurate nutritional information can be provided. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input data on the origin and production methods of the ingredients into a generating AI and use the nutrient calculation results output by the generating AI.

[0039] The analysis unit can provide analysis results while taking the user's allergy information into consideration during analysis. For example, if the user is allergic to a specific food, the unit will exclude the nutrients of dishes containing that food from the calculation. The analysis unit can also suggest alternative foods based on the user's allergy information. For example, it will suggest alternative foods based on the user's allergy information. The analysis unit can also display the risk of allergic reactions while taking the user's allergy information into consideration. For example, it will display the risk of allergic reactions while taking the user's allergy information into consideration. In this way, the risk of allergic reactions can be reduced by providing analysis results while taking the user's allergy information into consideration. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without using AI. For example, the analysis unit can input the user's allergy information data into a generating AI and use the analysis results output by the generating AI.

[0040] The analysis unit can customize the analysis results based on the user's health condition and goals during analysis. The analysis unit customizes the analysis results based on the user's health condition and goals during analysis, for example. For example, if the user is on a diet, the analysis unit can prioritize low-calorie ingredients. Furthermore, if the user is aiming to build muscle, the analysis unit can prioritize high-protein ingredients. For example, if the user is aiming to build muscle, the analysis unit can prioritize high-protein ingredients. Furthermore, if the user has diabetes, the analysis unit can prioritize low-carbohydrate ingredients. For example, if the user has diabetes, the analysis unit prioritizes low-carbohydrate ingredients. This allows for customizing the analysis results based on the user's health condition and goals, thereby providing more personalized information. Some or all of the above-described processing in the analysis unit may be performed using, or without, AI. For example, the analysis unit can input data on the user's health condition and goals into the generation AI and use the analysis results output by the generation AI.

[0041] The providing unit can provide optimal advice by referring to the user's past dietary history when providing the advice. For example, the providing unit can provide optimal advice by referring to the user's past dietary history when providing the advice. For example, if the user's past dietary history indicates an imbalance in nutrients, the providing unit can advise how to improve the balance. Furthermore, the providing unit can also provide advice on how to supplement a specific nutrient if the user is deficient in that nutrient based on the user's past dietary history. For example, if the user is deficient in a specific nutrient based on the user's past dietary history, the providing unit can advise how to supplement the nutrient. Furthermore, the providing unit can analyze the user's past dietary history and suggest healthy meals. For example, the providing unit can analyze the user's past dietary history and suggest healthy meals. By providing optimal advice by referring to the user's past dietary history, more effective advice can be provided. Some or all of the above-described processing by the providing unit can be performed using, for example, AI, or without AI. For example, the providing unit can input the user's past dietary history data into a generation AI and use the advice output by the generation AI.

[0042] The service provider can customize the content of the advice based on the user's health goals at the time of delivery. For example, if the user is on a diet, the service provider can suggest recipes using low-calorie ingredients. The service provider can also suggest recipes using high-protein ingredients if the user is aiming to build muscle. For example, if the user is aiming to build muscle, the service provider can suggest recipes using high-protein ingredients. The service provider can also suggest recipes using low-carbohydrate ingredients if the user has diabetes. For example, if the user has diabetes, the service provider can suggest recipes using low-carbohydrate ingredients. By customizing the content of the advice based on the user's health goals, more personalized advice can be provided. Some or all of the above processing in the service provider may be performed using AI, for example, or without AI. For example, the service provider can input the user's health goal data into a generating AI and use the advice content output by the generating AI.

[0043] The service provider can provide optimal advice by considering the user's geographical location at the time of delivery. For example, if the user is traveling, it can suggest recipes using local specialties. The service provider can also suggest healthy dishes from the menu if the user is in a restaurant. For example, if the user is in a restaurant, it can suggest healthy dishes from the menu. The service provider can also suggest recipes using ingredients that are readily available on a daily basis if the user is at home. For example, if the user is at home, it can suggest recipes using ingredients that are readily available on a daily basis. In this way, by providing optimal advice that considers the user's geographical location, it is possible to provide region-specific food information. Some or all of the above processing in the service provider may be performed using AI, for example, or without AI. For example, the service provider can input the user's geographical location data into a generating AI and use the advice output by the generating AI.

[0044] The providing unit can provide advice by analyzing the user's social media activity at the time of providing the advice. For example, the providing unit can provide advice by analyzing the user's social media activity at the time of providing the advice. For example, the providing unit can provide advice based on dishes shared by the user on social media. The providing unit can also provide advice based on dishes introduced by influencers followed by the user. For example, the providing unit can provide advice based on dishes introduced by influencers followed by the user. The providing unit can also provide advice based on dishes "liked" by the user. For example, the providing unit can provide advice based on dishes "liked" by the user. In this way, by analyzing the user's social media activity and providing advice, it is possible to provide advice that is tailored to the user's interests. Some or all of the above-described processing by the providing unit may be performed using, for example, AI, or may be performed without using AI. For example, the providing unit can input the user's social media activity data into a generation AI and use the advice output by the generation AI.

[0045] The history management unit can provide optimal advice by referring to the user's past dietary history during history management. For example, during history management, the history management unit can provide optimal advice by referring to the user's past dietary history. For example, if the user's past dietary history indicates an imbalance in nutrients, the history management unit can provide advice on how to improve the balance. The history management unit can also provide advice on how to supplement a specific nutrient based on the user's past dietary history if the user is deficient in that nutrient. For example, if the user's past dietary history indicates a nutrient deficiency, the history management unit can provide advice on how to supplement the nutrient. The history management unit can also analyze the user's past dietary history and suggest healthy meals. For example, the history management unit can analyze the user's past dietary history and suggest healthy meals. By providing optimal advice by referring to the user's past dietary history, more effective advice can be provided. Some or all of the above-described processing in the history management unit can be performed using, for example, AI, or without AI. For example, the history management unit can input the user's past dietary history data into a generation AI and use the advice output by the generation AI.

[0046] The history management unit can customize the history management method based on the user's health goals during history management. For example, the history management unit customizes the history management method based on the user's health goals during history management. For example, if the user is on a diet, the history management unit can highlight and display low-calorie meal history. Furthermore, if the user is aiming to build muscle, the history management unit can also highlight and display high-protein meal history. For example, if the user is aiming to build muscle, the history management unit can highlight and display low-carbohydrate meal history. For example, if the user is diabetic, the history management unit can highlight and display low-carbohydrate meal history. This allows for customizing the history management method based on the user's health goals, thereby providing more personalized information. Some or all of the above-described processing in the history management unit may be performed using, or without, AI. For example, the history management unit can input the user's health goal data into a generation AI and use the management method output by the generation AI.

[0047] The history management unit can provide an optimal history management method by taking into account the user's geographical location information during history management. For example, when managing history, the history management unit provides an optimal history management method by taking into account the user's geographical location information. For example, when a user is traveling, the history management unit can highlight and display meal histories that include local specialties. Furthermore, when a user is at a restaurant, the history management unit can highlight and display meal histories that include healthy dishes on the menu. For example, when a user is at a restaurant, the history management unit can highlight and display meal histories that include healthy dishes on the menu. Furthermore, when a user is at home, the history management unit can highlight and display meal histories that include commonly available ingredients. For example, when a user is at home, the history management unit highlights and displays meal histories that include commonly available ingredients. This provides an optimal history management method by taking into account the user's geographical location information, thereby enabling the provision of region-specific meal information. Some or all of the above-described processing in the history management unit may be performed using, or without, AI. For example, the history management unit can input the user's geographical location information data into a generation AI and use a management method output by the generation AI.

[0048] The history management unit can analyze a user's social media activity and manage the history during history management. For example, the history management unit can reflect dishes shared by a user on social media in the history. The history management unit can also reflect dishes introduced by influencers that the user follows in the history. For example, the history management unit can reflect dishes that a user has "liked" in the history. By analyzing a user's social media activity and managing the history, information tailored to the user's interests can be provided. Some or all of the above processing in the history management unit may be performed using AI, for example, or without AI. For example, the history management unit can input the user's social media activity data into a generating AI and use the history management method output by the generating AI.

[0049] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0050] The health management system can further acquire user exercise data and combine it with dietary analysis results to provide comprehensive health advice. For example, it can record the type and frequency of exercise a user performs daily, compare the nutritional balance of their diet with their exercise level, and suggest an appropriate calorie intake. Based on exercise data, it can also suggest recipes containing ingredients that may be needed to supplement specific nutrients. Furthermore, it can support the user's health maintenance by suggesting meals suitable for post-exercise recovery. In this way, the system can comprehensively manage the user's health from both dietary and exercise perspectives.

[0051] The analysis unit can analyze not only photos of the user's meals but also audio descriptions of the meals. For example, if the user describes the contents of the meal aloud, it can supplement the ingredients and cooking methods that are difficult to recognize from photos alone. Furthermore, by analyzing the audio data, detailed information about the user's meals can be obtained, allowing for more accurate nutrient calculations. Furthermore, the audio descriptions can be used to estimate the user's feelings and satisfaction with the meal and reflect this in the analysis results. This makes it possible to utilize both photos and audio to achieve more accurate dietary analysis.

[0052] The providing unit can automatically generate a shopping list of ingredients based on the results of the user's dietary analysis. For example, it can list ingredients to supplement nutrients that are lacking from the analysis results and suggest ingredients that the user should purchase the next time they go shopping. The providing unit can also customize the shopping list taking into account the user's preferences and allergy information. Furthermore, the providing unit can provide recipes that include the suggested ingredients, supporting the user in easily preparing healthy meals. This allows the user to efficiently purchase ingredients needed to improve their diet and maintain a healthy diet.

[0053] The history management unit can analyze seasonal dietary trends based on the user's dietary history and provide advice appropriate to the season. For example, since hydration is important in the summer, it can suggest ingredients and dishes that are high in water. It can also suggest ingredients that boost the immune system in the winter. Furthermore, the history management unit can analyze seasonal dietary trends and provide seasonal recipes based on the dishes the user has enjoyed in the past. This allows the user to enjoy healthy meals appropriate to the season.

[0054] The photographing unit can adjust the frequency of photographing specific ingredients or dishes by referring to the user's past meal history. For example, by prioritizing photographing ingredients or dishes that the user has photographed frequently in the past, the variety of meals can be increased. The photographing unit can also suggest ingredients or dishes that the user has not photographed much in the past, providing new meal options. Furthermore, the photographing unit can analyze the user's meal history and select ingredients and dishes to photograph taking nutritional balance into consideration. This allows the user to continuously record balanced meals and manage their health.

[0055] The camera can automatically adjust the background of the photo depending on the type and amount of food. For example, for large dishes, a simple background can be selected to highlight the food. For small dishes, a colorful background can be selected to balance the overall photo. Furthermore, if drinks are included, a specific theme can be set for the background to make the photo more appealing. This allows you to take more beautiful food photos by selecting the optimal background depending on the type and amount of food.

[0056] The photographing unit can emphasize regional ingredients and dishes by taking into account the user's geographical location information when photographing. For example, if the user is traveling, the photograph can be centered on local specialties. If the user is at a restaurant, the photograph can be emphasized on popular dishes on the menu. Furthermore, if the user is at home, the photograph can be emphasized on dishes that the user regularly eats. In this way, regional food information can be collected and provided to the user by taking into account the user's geographical location information.

[0057] The photography unit can analyze the user's social media activity when taking photos and suggest photography methods that reflect related food trends. For example, it can suggest photography methods for dishes introduced by influencers the user follows. It can also suggest similar compositions and filters based on the photography methods used to photograph dishes that the user has "liked." Furthermore, it can analyze the photography methods used to photograph dishes shared by the user and suggest more effective photography methods. This allows the system to suggest photography methods that reflect the user's social media activity, allowing the user to take photos of food that match their interests.

[0058] The processing flow of the first embodiment will be briefly explained below.

[0059] Step 1: The user takes a photo of their meal. The user does not need to perform any special operations; they simply take a photo of their meal with a camera such as a smartphone. For example, the user takes photos of breakfast, lunch, and dinner. This information is entered into the system. Step 2: The analysis unit analyzes the photo taken by the photography unit, recognizes the ingredients, and calculates the nutrients. The analysis unit uses, for example, AI to recognize the ingredients in the photo and calculate the nutrients for each. For example, it identifies the vegetables, meat, fish, and other ingredients in the photo and calculates the calories and nutrients for each. Step 3: The provider provides advice and recommendations on nutrient balance, calories, and dietary improvements based on the analysis results obtained by the analyzer. For example, if the nutrient balance is unbalanced based on a photo taken by the user, the provider provides advice on how to improve the diet. Step 4: The history management unit manages the user's dietary history based on the information provided by the provision unit, compares it with past data, and provides advice. For example, if there is an imbalance in nutrients based on the dietary data from the past week, advice will be given on how to improve it.

[0060] (Example 2) A health management system according to an embodiment of the present invention analyzes a user's meal content and provides advice and advice on nutrient balance and calories, as well as suggestions for improving the diet, simply by the user taking a photo of the meal. This health management system allows the user to take a photo of the meal, analyze the photo, recognize ingredients, and calculate nutrients. Furthermore, based on the analysis results, the system provides advice and advice on nutrient balance and calories, as well as suggestions for improving the diet. For example, a user simply takes a photo of their breakfast, lunch, and dinner, and the system automatically analyzes the meal content, calculates the nutrient balance and calories, and provides these to the user. The system also manages the user's diet history and compares it with past data to provide advice. For example, if there is an imbalance in nutrients based on the diet data from the past week, the system provides advice on how to improve the diet. This allows the user to easily manage their diet. Thus, the health management system allows the user to analyze a meal content and provide advice and advice on nutrient balance and calories, as well as suggestions for improving the diet, simply by the user taking a photo of the meal.

[0061] A health management system according to an embodiment includes a photographing unit, an analysis unit, a providing unit, and a history management unit. The photographing unit allows a user to take photos of their meals. When taking photos of meals, the user simply takes photos of the meals with a camera such as a smartphone without any special operations. For example, the user takes photos of breakfast, lunch, and dinner. This information is input into the system. The analysis unit analyzes the photos taken by the photographing unit, recognizes ingredients, and calculates the nutritional value of each. The analysis unit, for example, uses AI to recognize ingredients in the photos and calculates the nutritional value of each. For example, the analysis unit identifies ingredients such as vegetables, meat, and fish in the photos and calculates the calories and nutrients of each. The providing unit provides the user with information on nutrient balance and calories, as well as advice and suggestions for improving their diet, based on the analysis results obtained by the analysis unit. For example, the providing unit provides the user with information on nutrient balance and calories, as well as advice and suggestions for improving their diet. For example, if the user's diet is unbalanced based on photos taken by the user, the providing unit provides advice on how to improve the diet. The history management unit manages the user's diet history based on the information provided by the providing unit and compares it with past data to provide advice. The history management unit, for example, manages the user's diet history and compares it with past data to provide advice. For example, if the user's nutritional balance is unbalanced based on the dietary data from the past week, advice on how to improve the balance can be provided. This allows the health management system according to the embodiment to analyze the contents of a meal and provide advice on the balance of nutrients, calories, and how to improve the diet simply by the user taking a photo of the meal.

[0062] The analysis unit can recognize ingredients in a photo and calculate the nutritional value of each ingredient. The analysis unit can use, for example, AI to recognize ingredients in a photo and calculate the nutritional value of each ingredient. For example, it can identify ingredients such as vegetables, meat, and fish in the photo and calculate the calories and nutrients of each ingredient. The analysis unit can also use an image recognition algorithm to recognize ingredients and calculate nutrients by referencing a database. For example, the image recognition algorithm can identify ingredients in the photo and obtain nutrient information for those ingredients from a database for calculation. This allows accurate nutritional information to be provided by recognizing ingredients in the photo and calculating their nutritional value. Some or all of the above-described processing in the analysis unit can be performed using, for example, AI, or without AI. For example, the analysis unit can input the recognition of ingredients in the photo and the calculation of nutrients into an AI model and use the analysis results output by the AI ​​model.

[0063] The providing unit can provide nutrient balance, calories, and dietary improvements or advice. The providing unit provides the user with nutrient balance, calories, dietary improvements, and advice based on, for example, the analysis results obtained by the analysis unit. For example, if the nutrient balance is unbalanced based on a photo taken by the user, the providing unit can provide advice on how to improve the nutrient balance. The providing unit can also calculate nutrient balance and calories and provide the user with the same. For example, based on the nutrient balance and calories calculated by the analysis unit, the providing unit can provide the user with dietary improvements or advice. This can help the user improve their eating habits by providing nutrient balance, calories, dietary improvements, and advice. Some or all of the above-described processing by the providing unit can be performed using, for example, AI, or can be performed without using AI. For example, the providing unit can input the analysis results obtained by the analysis unit into an AI model and use the advice output by the AI ​​model.

[0064] The history management unit can manage the user's diet history and provide advice by comparing it with past data. For example, the history management unit can manage the user's diet history and provide advice by comparing it with past data. For example, if there is an imbalance in nutrients based on diet data from the past week, the history management unit can provide advice on how to improve the imbalance. The history management unit can also store the user's diet history in a database and provide advice by comparing it with past data. For example, the history management unit can store the user's diet history in a database and provide advice by comparing it with past data to analyze nutrient balance and calorie fluctuations. This allows the user to continuously improve their diet by managing their diet history and providing advice by comparing it with past data. Some or all of the above-described processing in the history management unit can be performed using, for example, AI, or without AI. For example, the history management unit can input the user's diet history into an AI model and use the advice output by the AI ​​model.

[0065] The photographing unit can estimate the user's emotions and adjust the timing of photographing based on the estimated user emotions. For example, the photographing unit estimates the user's emotions and adjusts the timing of photographing based on the estimated user emotions. For example, if the user is relaxed, photographing can automatically start when the meal is ready. Furthermore, if the user is in a hurry, the photographing unit can adjust the timing to quickly photograph the entire meal. For example, if the user is in a hurry, the photographing unit quickly photographs the entire meal. Furthermore, if the user is feeling stressed, the photographing unit can flexibly adjust the timing of photographing to match the user's pace. For example, if the user is feeling stressed, the photographing unit flexibly adjusts the timing of photographing to match the user's pace. This allows the photographing timing to be adjusted according to the user's emotions, thereby enabling the photograph of the meal to be taken at a more appropriate time. Emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the photographing unit may be performed using, for example, AI, or without AI. For example, the photographing unit can input the user's emotional data into the generation AI and use the photographing timing output by the generation AI.

[0066] The photographing unit can suggest the optimal photographing method by referring to the user's past meal history when photographing. For example, the photographing unit can suggest the optimal photographing method by referring to the user's past meal history when photographing. For example, the photographing unit can analyze photos of meals taken by the user in the past and suggest the most effective photographing angle. The photographing unit can also adjust the photographing method based on the user's past meal history so that specific ingredients are best captured. For example, the photographing unit can adjust the photographing method based on the user's past meal history so that specific ingredients are best captured. The photographing unit can also optimize the lighting conditions during photographing based on the user's past meal history. For example, the lighting conditions during photographing are optimized based on the user's past meal history. This allows for more effective photographs of food to be taken by suggesting the optimal photographing method by referring to the user's past meal history. Some or all of the above-described processing in the photographing unit may be performed using, for example, AI, or may be performed without AI. For example, the photographing unit can input the user's past meal history data into a generation AI and use the optimal photographing method output by the generation AI.

[0067] The camera unit can automatically switch the shooting mode depending on the type and amount of food when taking a photo. The camera unit, for example, automatically switches the shooting mode depending on the type and amount of food when taking a photo. For example, when a large dish is being served, the camera unit switches to wide-angle mode to capture the entire dish. In addition, when a small dish is being served, the camera unit can switch to macro mode to capture the details. For example, when a small dish is being served, the camera unit switches to macro mode to capture the details. In addition, when a drink is included, the camera unit can apply a specific filter to emphasize the color. For example, when a drink is included, the camera unit applies a specific filter to emphasize the color. In this way, the optimal photo can be taken by automatically switching the shooting mode depending on the type and amount of food. Some or all of the above-mentioned processing in the camera unit may be performed using, or without, AI. For example, the camera unit can input data on the type and amount of food into the generation AI and use the shooting mode output by the generation AI.

[0068] The camera unit can estimate the user's emotions and determine the priority of meals to photograph based on the estimated emotions. For example, if the user is enjoying themselves, special dishes such as desserts will be photographed first. The camera unit can also prioritize photographing main dishes if the user is tired. For example, if the user is tired, main dishes will be photographed first. The camera unit can also prioritize photographing vegetables and fruits if the user is health-conscious. For example, if the user is health-conscious, vegetables and fruits will be photographed first. This allows important meals to be photographed first by determining the priority of meals to photograph according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the camera unit may be performed using AI, for example, or without AI. For example, the photographing unit can input the user's emotional data into the generation AI and use the photographing priority order output by the generation AI.

[0069] The photographing unit can prioritize photographing highly relevant meals in consideration of the user's geographical location information when photographing. For example, the photographing unit prioritizes photographing highly relevant meals in consideration of the user's geographical location information when photographing. For example, when the user is traveling, the photographing unit prioritizes photographing local specialties. Furthermore, when the user is at a restaurant, the photographing unit can prioritize photographing popular dishes on the menu. For example, when the user is at a restaurant, the photographing unit prioritizes photographing popular dishes on the menu. Furthermore, when the user is at home, the photographing unit can prioritize photographing dishes that the user eats on a daily basis. For example, when the user is at home, the photographing unit prioritizes photographing dishes that the user eats on a daily basis. In this way, by prioritizing photographing highly relevant meals in consideration of the user's geographical location information, it is possible to collect region-specific meal information. Some or all of the above-described processing by the photographing unit may be performed using, for example, AI, or may be performed without using AI. For example, the photographing unit can input the user's geographical location information data into a generation AI and use the highly relevant meals output by the generation AI.

[0070] The photography unit can analyze the user's social media activity during photography and photograph related meals. For example, the photography unit can prioritize photographing dishes that the user has shared on social media. The photography unit can also prioritize photographing dishes that have been featured by influencers that the user follows. For example, the photography unit can prioritize photographing dishes that have been featured by influencers that the user follows. The photography unit can also prioritize photographing dishes that the user has "liked". For example, the photography unit can prioritize photographing dishes that have been "liked". By analyzing the user's social media activity and photographing related meals, it is possible to collect food information that matches the user's interests. Some or all of the above processing in the photography unit may be performed using AI, for example, or without AI. For example, the photography unit can input the user's social media activity data into a generating AI and use the related meals output by the generating AI.

[0071] The analysis unit can estimate the user's emotions and adjust the accuracy of the analysis based on the estimated emotions. For example, if the user is relaxed, the analysis unit can perform a detailed analysis and calculate the fine balance of nutrients. The analysis unit can also quickly analyze only the major nutrients if the user is in a hurry. For example, if the user is in a hurry, it can quickly analyze only the major nutrients. The analysis unit can also provide a simplified analysis result if the user is stressed. For example, if the user is stressed, it can provide a simplified analysis result. By adjusting the accuracy of the analysis according to the user's emotions, more appropriate analysis results can be provided. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input the user's emotional data into the generation AI and use the analysis accuracy output by the generation AI.

[0072] The analysis unit can calculate nutrients while considering the freshness and cooking method of the ingredients during analysis. For example, it can calculate nutrients while considering the difference in nutrients between fresh vegetables and frozen vegetables. The analysis unit can also calculate nutrients while considering the difference in nutrients between grilled fish and raw fish. For example, it can calculate nutrients while considering the difference in nutrients between grilled fish and raw fish. The analysis unit can also calculate nutrients while considering the difference in nutrients between dishes with long cooking times and dishes with short cooking times. For example, it can calculate nutrients while considering the difference in nutrients between dishes with long cooking times and dishes with short cooking times. By calculating nutrients while considering the freshness and cooking method of the ingredients, more accurate nutritional information can be provided. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input data on the freshness and cooking method of the ingredients into a generating AI and use the nutrient calculation results output by the generating AI.

[0073] The analysis unit can calculate nutrients while considering the origin and production methods of the ingredients during analysis. For example, it can calculate nutrients while considering the differences in nutrients between organically grown vegetables and conventionally grown vegetables. The analysis unit can also calculate nutrients while considering the differences in nutrients between locally produced and imported ingredients. For example, it can calculate nutrients while considering the differences in nutrients between locally produced and imported ingredients. The analysis unit can also calculate nutrients while considering the differences in nutrients between meat from free-range cattle and meat from factory-produced cattle. For example, it can calculate nutrients while considering the differences in nutrients between meat from free-range cattle and meat from factory-produced cattle. By calculating nutrients while considering the origin and production methods of the ingredients, more accurate nutritional information can be provided. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without AI. For example, the analysis unit can input data on the origin and production methods of the ingredients into a generating AI and use the nutrient calculation results output by the generating AI.

[0074] The analysis unit can estimate the user's emotions and adjust the display method of the analysis results based on the estimated user emotions. For example, the analysis unit estimates the user's emotions and adjusts the display method of the analysis results based on the estimated user emotions. For example, if the user is nervous, the analysis unit provides a simple, highly visible display method. Furthermore, if the user is relaxed, the analysis unit can also provide a display method including detailed information. For example, if the user is relaxed, the analysis unit provides a display method including detailed information. Furthermore, if the user is in a hurry, the analysis unit can also provide a display method that focuses on the main points. For example, if the user is in a hurry, the analysis unit provides a display method that focuses on the main points. This allows for adjusting the display method of the analysis results according to the user's emotions, thereby providing more appropriate information. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to these examples. Some or all of the above-described processing in the analysis unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the analysis unit can input user emotion data into a generating AI and use the display method output by the generating AI.

[0075] The analysis unit can provide analysis results while taking the user's allergy information into consideration during analysis. For example, if the user is allergic to a specific food, the unit will exclude the nutrients of dishes containing that food from the calculation. The analysis unit can also suggest alternative foods based on the user's allergy information. For example, it will suggest alternative foods based on the user's allergy information. The analysis unit can also display the risk of allergic reactions while taking the user's allergy information into consideration. For example, it will display the risk of allergic reactions while taking the user's allergy information into consideration. In this way, the risk of allergic reactions can be reduced by providing analysis results while taking the user's allergy information into consideration. Some or all of the above processing in the analysis unit may be performed using AI, for example, or without using AI. For example, the analysis unit can input the user's allergy information data into a generating AI and use the analysis results output by the generating AI.

[0076] The analysis unit can customize the analysis results based on the user's health condition and goals during analysis. The analysis unit customizes the analysis results based on the user's health condition and goals during analysis, for example. For example, if the user is on a diet, the analysis unit can prioritize low-calorie ingredients. Furthermore, if the user is aiming to build muscle, the analysis unit can prioritize high-protein ingredients. For example, if the user is aiming to build muscle, the analysis unit can prioritize high-protein ingredients. Furthermore, if the user has diabetes, the analysis unit can prioritize low-carbohydrate ingredients. For example, if the user has diabetes, the analysis unit prioritizes low-carbohydrate ingredients. This allows for customizing the analysis results based on the user's health condition and goals, thereby providing more personalized information. Some or all of the above-described processing in the analysis unit may be performed using, or without, AI. For example, the analysis unit can input data on the user's health condition and goals into the generation AI and use the analysis results output by the generation AI.

[0077] The service provider can estimate the user's emotions and adjust the content of the advice provided based on the estimated emotions. For example, if the user is relaxed, it can provide detailed advice. The service provider can also provide concise advice if the user is in a hurry. For example, if the user is in a hurry, it can provide concise advice. The service provider can also provide advice that includes words of encouragement if the user is stressed. For example, if the user is stressed, it can provide advice that includes words of encouragement. By adjusting the content of the advice according to the user's emotions, more appropriate advice can be provided. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above processing in the service provider may be performed using AI, for example, or without AI. For example, the service provider can input the user's emotion data into a generative AI and use the advice content output by the generative AI.

[0078] The providing unit can provide optimal advice by referring to the user's past dietary history when providing the advice. For example, the providing unit can provide optimal advice by referring to the user's past dietary history when providing the advice. For example, if the user's past dietary history indicates an imbalance in nutrients, the providing unit can advise how to improve the balance. Furthermore, the providing unit can also provide advice on how to supplement a specific nutrient if the user is deficient in that nutrient based on the user's past dietary history. For example, if the user is deficient in a specific nutrient based on the user's past dietary history, the providing unit can advise how to supplement the nutrient. Furthermore, the providing unit can analyze the user's past dietary history and suggest healthy meals. For example, the providing unit can analyze the user's past dietary history and suggest healthy meals. By providing optimal advice by referring to the user's past dietary history, more effective advice can be provided. Some or all of the above-described processing by the providing unit can be performed using, for example, AI, or without AI. For example, the providing unit can input the user's past dietary history data into a generation AI and use the advice output by the generation AI.

[0079] The service provider can customize the content of the advice based on the user's health goals at the time of delivery. For example, if the user is on a diet, the service provider can suggest recipes using low-calorie ingredients. The service provider can also suggest recipes using high-protein ingredients if the user is aiming to build muscle. For example, if the user is aiming to build muscle, the service provider can suggest recipes using high-protein ingredients. The service provider can also suggest recipes using low-carbohydrate ingredients if the user has diabetes. For example, if the user has diabetes, the service provider can suggest recipes using low-carbohydrate ingredients. By customizing the content of the advice based on the user's health goals, more personalized advice can be provided. Some or all of the above processing in the service provider may be performed using AI, for example, or without AI. For example, the service provider can input the user's health goal data into a generating AI and use the advice content output by the generating AI.

[0080] The providing unit can estimate the user's emotions and determine the priority of advice based on the estimated user emotions. The providing unit, for example, estimates the user's emotions and determines the priority of advice based on the estimated user emotions. For example, if the user is relaxed, detailed advice can be provided preferentially. Furthermore, if the user is in a hurry, the providing unit can also provide concise advice preferentially. For example, if the user is in a hurry, concise advice can be provided preferentially. Furthermore, if the user is feeling stressed, the providing unit can also provide advice including encouraging words preferentially. For example, if the user is feeling stressed, the providing unit can provide advice including encouraging words preferentially. In this way, by determining the priority of advice according to the user's emotions, more appropriate advice can be provided. Emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI can be, for example, a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the providing unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the providing unit can input the user's emotional data into the generating AI and use the advice priority output by the generating AI.

[0081] The service provider can provide optimal advice by considering the user's geographical location at the time of delivery. For example, if the user is traveling, it can suggest recipes using local specialties. The service provider can also suggest healthy dishes from the menu if the user is in a restaurant. For example, if the user is in a restaurant, it can suggest healthy dishes from the menu. The service provider can also suggest recipes using ingredients that are readily available on a daily basis if the user is at home. For example, if the user is at home, it can suggest recipes using ingredients that are readily available on a daily basis. In this way, by providing optimal advice that considers the user's geographical location, it is possible to provide region-specific food information. Some or all of the above processing in the service provider may be performed using AI, for example, or without AI. For example, the service provider can input the user's geographical location data into a generating AI and use the advice output by the generating AI.

[0082] The providing unit can provide advice by analyzing the user's social media activity at the time of providing the advice. For example, the providing unit can provide advice by analyzing the user's social media activity at the time of providing the advice. For example, the providing unit can provide advice based on dishes shared by the user on social media. The providing unit can also provide advice based on dishes introduced by influencers followed by the user. For example, the providing unit can provide advice based on dishes introduced by influencers followed by the user. The providing unit can also provide advice based on dishes "liked" by the user. For example, the providing unit can provide advice based on dishes "liked" by the user. In this way, by analyzing the user's social media activity and providing advice, it is possible to provide advice that is tailored to the user's interests. Some or all of the above-described processing by the providing unit may be performed using, for example, AI, or may be performed without using AI. For example, the providing unit can input the user's social media activity data into a generation AI and use the advice output by the generation AI.

[0083] The history management unit can estimate the user's emotions and adjust the display method of the history based on the estimated user emotions. For example, the history management unit estimates the user's emotions and adjusts the display method of the history based on the estimated user emotions. For example, if the user is relaxed, a detailed history is displayed. Furthermore, if the user is in a hurry, the history management unit can display a history that emphasizes the main points. For example, if the user is in a hurry, a history that emphasizes the main points is displayed. Furthermore, if the user is feeling stressed, the history management unit can display a simplified history. For example, if the user is feeling stressed, a simplified history is displayed. In this way, by adjusting the display method of the history according to the user's emotions, more appropriate information can be provided. Emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI can be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to these examples. Some or all of the above-mentioned processing in the history management unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the history management unit can input user emotion data into a generating AI and use the display method output by the generating AI.

[0084] The history management unit can provide optimal advice by referring to the user's past dietary history during history management. For example, during history management, the history management unit can provide optimal advice by referring to the user's past dietary history. For example, if the user's past dietary history indicates an imbalance in nutrients, the history management unit can provide advice on how to improve the balance. The history management unit can also provide advice on how to supplement a specific nutrient based on the user's past dietary history if the user is deficient in that nutrient. For example, if the user's past dietary history indicates a nutrient deficiency, the history management unit can provide advice on how to supplement the nutrient. The history management unit can also analyze the user's past dietary history and suggest healthy meals. For example, the history management unit can analyze the user's past dietary history and suggest healthy meals. By providing optimal advice by referring to the user's past dietary history, more effective advice can be provided. Some or all of the above-described processing in the history management unit can be performed using, for example, AI, or without AI. For example, the history management unit can input the user's past dietary history data into a generation AI and use the advice output by the generation AI.

[0085] The history management unit can customize the history management method based on the user's health goals during history management. For example, the history management unit customizes the history management method based on the user's health goals during history management. For example, if the user is on a diet, the history management unit can highlight and display low-calorie meal history. Furthermore, if the user is aiming to build muscle, the history management unit can also highlight and display high-protein meal history. For example, if the user is aiming to build muscle, the history management unit can highlight and display low-carbohydrate meal history. For example, if the user is diabetic, the history management unit can highlight and display low-carbohydrate meal history. This allows for customizing the history management method based on the user's health goals, thereby providing more personalized information. Some or all of the above-described processing in the history management unit may be performed using, or without, AI. For example, the history management unit can input the user's health goal data into a generation AI and use the management method output by the generation AI.

[0086] The history management unit can estimate the user's emotions and prioritize the history based on the estimated user emotions. The history management unit, for example, estimates the user's emotions and prioritizes the history based on the estimated user emotions. For example, if the user is relaxed, the history management unit can prioritize displaying detailed history. Furthermore, if the user is in a hurry, the history management unit can prioritize displaying history that emphasizes the main points. For example, if the user is in a hurry, the history management unit can prioritize displaying history that emphasizes the main points. Furthermore, if the user is feeling stressed, the history management unit can prioritize displaying simple history. For example, if the user is feeling stressed, the history management unit can prioritize displaying simple history. In this way, by prioritizing the history based on the user's emotions, more appropriate information can be provided. Emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the history management unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the history management unit can input the user's emotional data into the generation AI and use the priority order output by the generation AI.

[0087] The history management unit can provide an optimal history management method by taking into account the user's geographical location information during history management. For example, when managing history, the history management unit provides an optimal history management method by taking into account the user's geographical location information. For example, when a user is traveling, the history management unit can highlight and display meal histories that include local specialties. Furthermore, when a user is at a restaurant, the history management unit can highlight and display meal histories that include healthy dishes on the menu. For example, when a user is at a restaurant, the history management unit can highlight and display meal histories that include healthy dishes on the menu. Furthermore, when a user is at home, the history management unit can highlight and display meal histories that include commonly available ingredients. For example, when a user is at home, the history management unit highlights and displays meal histories that include commonly available ingredients. This provides an optimal history management method by taking into account the user's geographical location information, thereby enabling the provision of region-specific meal information. Some or all of the above-described processing in the history management unit may be performed using, or without, AI. For example, the history management unit can input the user's geographical location information data into a generation AI and use a management method output by the generation AI.

[0088] The history management unit can analyze a user's social media activity and manage the history during history management. For example, the history management unit can reflect dishes shared by a user on social media in the history. The history management unit can also reflect dishes introduced by influencers that the user follows in the history. For example, the history management unit can reflect dishes that a user has "liked" in the history. By analyzing a user's social media activity and managing the history, information tailored to the user's interests can be provided. Some or all of the above processing in the history management unit may be performed using AI, for example, or without AI. For example, the history management unit can input the user's social media activity data into a generating AI and use the history management method output by the generating AI. === Hard Collateral 1-1 === Each of the multiple elements, including the above-mentioned photographing unit, analysis unit, provision unit, and history management unit, is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the photographing unit is realized by the camera 42 of the smart device 14, and the user takes a photo of the meal. The analysis unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and uses AI to recognize ingredients in the photo and calculate the nutrients of each. The provision unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and provides the user with information on nutrient balance, calories, dietary improvements, and advice based on the analysis results. The history management unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and manages the user's diet history and provides advice by comparing it with past data. === Hard Collateral 1-2 === Each of the multiple elements described above, including the shooting unit, analysis unit, provision unit, and history management unit, is implemented, for example, in at least one of the smart glasses 214 and the data processing unit 12. For example, the shooting unit is implemented by the camera 42 of the smart glasses 214, which takes a picture of the meal taken by the user. The analysis unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which uses AI to recognize the ingredients in the picture and calculate the nutrients of each. The provision unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which provides the user with information on the balance of nutrients, calories, and suggestions and advice for improving the meal based on the analysis results. The history management unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which manages the user's meal history and provides advice by comparing it with past data. === Hard Collateral 1-3 === Each of the multiple elements described above, including the shooting unit, analysis unit, provision unit, and history management unit, is implemented, for example, by at least one of the headset terminal 314 and the data processing unit 12. For example, the shooting unit is implemented by the camera 42 of the headset terminal 314, which takes a picture of the meal. The analysis unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which uses AI to recognize the ingredients in the picture and calculate the nutrients of each. The provision unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which provides the user with information on the balance of nutrients, calories, and suggestions and advice for improving the meal based on the analysis results. The history management unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12, which manages the user's meal history and provides advice by comparing it with past data. === Hard Collateral 1-4 === Each of the multiple elements including the above-mentioned photographing unit, analysis unit, provision unit, and history management unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the photographing unit is realized by the camera 42 of the robot 414, and the user takes a photo of the meal. The analysis unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and uses AI to recognize ingredients in the photo and calculate the nutrients of each. The provision unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and provides the user with information on nutrient balance, calories, dietary improvements, and advice based on the analysis results. The history management unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and manages the user's dietary history and provides advice by comparing it with past data.

[0089] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0090] The health management system can further acquire user exercise data and combine it with dietary analysis results to provide comprehensive health advice. For example, it can record the type and frequency of exercise a user performs daily, compare the nutritional balance of their diet with their exercise level, and suggest an appropriate calorie intake. Based on exercise data, it can also suggest recipes containing ingredients that may be needed to supplement specific nutrients. Furthermore, it can support the user's health maintenance by suggesting meals suitable for post-exercise recovery. In this way, the system can comprehensively manage the user's health from both dietary and exercise perspectives.

[0091] The analysis unit can analyze not only photos of the user's meals but also audio descriptions of the meals. For example, if the user describes the contents of the meal aloud, it can supplement the ingredients and cooking methods that are difficult to recognize from photos alone. Furthermore, by analyzing the audio data, detailed information about the user's meals can be obtained, allowing for more accurate nutrient calculations. Furthermore, the audio descriptions can be used to estimate the user's feelings and satisfaction with the meal and reflect this in the analysis results. This makes it possible to utilize both photos and audio to achieve more accurate dietary analysis.

[0092] The providing unit can automatically generate a shopping list of ingredients based on the results of the user's dietary analysis. For example, it can list ingredients to supplement nutrients that are lacking from the analysis results and suggest ingredients that the user should purchase the next time they go shopping. The providing unit can also customize the shopping list taking into account the user's preferences and allergy information. Furthermore, the providing unit can provide recipes that include the suggested ingredients, supporting the user in easily preparing healthy meals. This allows the user to efficiently purchase ingredients needed to improve their diet and maintain a healthy diet.

[0093] The history management unit can analyze seasonal dietary trends based on the user's dietary history and provide advice appropriate to the season. For example, since hydration is important in the summer, it can suggest ingredients and dishes that are high in water. It can also suggest ingredients that boost the immune system in the winter. Furthermore, the history management unit can analyze seasonal dietary trends and provide seasonal recipes based on the dishes the user has enjoyed in the past. This allows the user to enjoy healthy meals appropriate to the season.

[0094] The photographing unit can estimate the user's emotions and automatically apply a filter during photographing based on the estimated user's emotions. For example, if the user is relaxed, a warm-colored filter can be applied to make the meal photo look more attractive. If the user is stressed, a cool-colored filter can be applied to create a calm atmosphere. Furthermore, if the user is having fun, a vibrant-colored filter can be applied to make the meal photo look more lively. In this way, by applying a filter according to the user's emotions, it is possible to take more attractive photographs of meals.

[0095] The photographing unit can adjust the frequency of photographing specific ingredients or dishes by referring to the user's past meal history. For example, by prioritizing photographing ingredients or dishes that the user has photographed frequently in the past, the variety of meals can be increased. The photographing unit can also suggest ingredients or dishes that the user has not photographed much in the past, providing new meal options. Furthermore, the photographing unit can analyze the user's meal history and select ingredients and dishes to photograph taking nutritional balance into consideration. This allows the user to continuously record balanced meals and manage their health.

[0096] The camera can automatically adjust the background of the photo depending on the type and amount of food. For example, for large dishes, a simple background can be selected to highlight the food. For small dishes, a colorful background can be selected to balance the overall photo. Furthermore, if drinks are included, a specific theme can be set for the background to make the photo more appealing. This allows you to take more beautiful food photos by selecting the optimal background depending on the type and amount of food.

[0097] The photographing unit can estimate the user's emotions and adjust the composition of the meal to be photographed based on the estimated user's emotions. For example, if the user is relaxed, a wide-angle composition can be selected to capture the entire meal. If the user is in a hurry, a composition can be selected to capture a close-up of the main dish. Furthermore, if the user is enjoying themselves, a composition can be selected to focus on desserts or special dishes. In this way, by selecting a composition that corresponds to the user's emotions, more attractive meal photos can be taken.

[0098] The photographing unit can emphasize regional ingredients and dishes by taking into account the user's geographical location information when photographing. For example, if the user is traveling, the photograph can be centered on local specialties. If the user is at a restaurant, the photograph can be emphasized on popular dishes on the menu. Furthermore, if the user is at home, the photograph can be emphasized on dishes that the user regularly eats. In this way, regional food information can be collected and provided to the user by taking into account the user's geographical location information.

[0099] The photography unit can analyze the user's social media activity when taking photos and suggest photography methods that reflect related food trends. For example, it can suggest photography methods for dishes introduced by influencers the user follows. It can also suggest similar compositions and filters based on the photography methods used to photograph dishes that the user has "liked." Furthermore, it can analyze the photography methods used to photograph dishes shared by the user and suggest more effective photography methods. This allows the system to suggest photography methods that reflect the user's social media activity, allowing the user to take photos of food that match their interests.

[0100] The processing flow of the second embodiment will be briefly explained below.

[0101] Step 1: The user takes a photo of their meal. The user does not need to perform any special operations; they simply take a photo of their meal with a camera such as a smartphone. For example, the user takes photos of breakfast, lunch, and dinner. This information is entered into the system. Step 2: The analysis unit analyzes the photo taken by the photography unit, recognizes the ingredients, and calculates the nutrients. The analysis unit uses, for example, AI to recognize the ingredients in the photo and calculate the nutrients for each. For example, it identifies the vegetables, meat, fish, and other ingredients in the photo and calculates the calories and nutrients for each. Step 3: The provider provides advice and recommendations on nutrient balance, calories, and dietary improvements based on the analysis results obtained by the analyzer. For example, if the nutrient balance is unbalanced based on a photo taken by the user, the provider provides advice on how to improve the diet. Step 4: The history management unit manages the user's dietary history based on the information provided by the provision unit, compares it with past data, and provides advice. For example, if there is an imbalance in nutrients based on the dietary data from the past week, advice will be given on how to improve it.

[0102] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of the generative AI include a neural network (NN) and a neural network (NN). The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, image data, etc. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and may perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-mentioned parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. The processing performed by an AI including the generative AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI including the generative AI.

[0104] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0105] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0106] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0107] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0108] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0109] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0110] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0111] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0112] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0113] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0114] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0115] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0116] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0117] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0118] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0119] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0120] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0121] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0122] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0123] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0124] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0125] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0126] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0127] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0128] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0129] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0130] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0131] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0132] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0133] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0134] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0135] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0136] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0137] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0138] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0139] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0140] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0141] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0142] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0143] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0144] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0145] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0146] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0147] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0149] In the robot 414, the processor 46 performs the identification process. The storage 50 stores the identification program 60. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as the control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform the same process as the identification processing unit 290 using these models.

[0150] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0151] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0152] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0153] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0154] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0155] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0156] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0157] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0158] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0159] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0160] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0161] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0162] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0163] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0164] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0165] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0166] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0167] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0168] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0169] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0170] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0171] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0172] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0173] [Explanation of symbols]

[0174] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a photographing unit for allowing a user to take a photograph of a meal; an analysis unit that analyzes the photograph taken by the photographing unit, recognizes ingredients, and calculates nutrients; a providing unit that provides advice or improvements on nutrient balance, calories, and diet based on the analysis results obtained by the analyzing unit; a history management unit that manages the user's diet history based on the information provided by the providing unit, compares it with past data, and provides advice. A system characterized by:

2. The analysis unit Recognizes ingredients in a photo and calculates the nutritional value of each 2. The system of claim 1.

3. The providing unit Providing nutritional balance, calorie and dietary improvements or advice 2. The system of claim 1.

4. The history management unit Manages the user's dietary history and provides advice by comparing it with past data 2. The system of claim 1.

5. The imaging unit is Estimate the user's emotions and adjust the timing of taking photos based on the estimated user emotions.

2. The system of claim 1.

6. The imaging unit is When taking a photo, the system refers to the user's past meal history and suggests the optimal shooting method.

2. The system of claim 1.

7. The imaging unit is When taking a photo, the camera automatically switches the shooting mode depending on the type and amount of food.

2. The system of claim 1.

8. The imaging unit is Estimate the user's emotions and prioritize the food to be photographed based on the estimated user emotions.

2. The system of claim 1.

Images