system

Abstract

We provide the system. [Solution] A means of receiving user posts, An identification method for identifying a dish from an image included in a received post, A means of calculating nutritional balance based on a specific dish, A method for selecting dishes to be proposed based on a calculated nutritional balance, A means of notifying the user of the above proposed dishes, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, many people aim for a healthy diet, but it is not easy to accurately grasp what nutritional balance their diet has. Also, identifying nutrients lacking in daily meals and selecting appropriate dishes and ingredients to supplement them has become a significant burden. In such a situation, there is a need to provide means for automatically obtaining nutritional information from cooking photos included in users' SNS posts and making specific proposals.

Means for Solving the Problems

[0005] This invention solves this problem by providing a system that analyzes images of dishes posted by users on social media. The system receives a dish image, uses image recognition technology to identify the dish name and related ingredient information, and calculates the nutritional balance. Based on the calculated nutritional data, it identifies the nutrients the user is lacking, automatically selects suggested dishes to supplement those deficiencies, and notifies the user. This allows users to be mindful of nutritional balance in their daily meals and achieve appropriate nutritional intake without any extra effort.

[0006] A "user" refers to an individual who uses the system to post images of their dishes and receives nutritional information analysis and suggestions.

[0007] "Post" refers to content uploaded by users to social media, and in this invention in particular, it refers to images of food and related information.

[0008] "Images" refer to visual data posted by users, such as photographs and visual information that show the contents of a dish.

[0009] "Cooking" refers to food or prepared meals included in an image, and the subject from which nutritional information can be extracted through analysis.

[0010] "Identification means" refers to functions and methods for identifying the type of dish and ingredients from posted images using image recognition technology.

[0011] "Nutritional balance" refers to the proportion and arrangement of each nutrient contained in a dish or meal, and describes the combination of nutrients necessary for good health.

[0012] "Suggested dishes" refer to dishes or ingredients selected by the system to improve the user's nutritional balance.

[0013] "Notification" refers to a means of communicating suggestions to the user, and includes formats such as in-app messages and push notifications.

[0014] "Store information" refers to data and details about the locations where users can actually obtain the dishes suggested to them. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that automatically analyzes nutritional information from food images posted by users on social media and suggests healthy meals. This system is implemented in the following specific form.

[0037] First, users link their social media accounts with this system. This allows the server to efficiently retrieve food photos that have been publicly shared from the user's account. The server periodically or in response to user actions uses social media APIs to collect posted image data.

[0038] Next, the server uses an AI module equipped with image recognition technology to analyze the acquired images. The analysis process identifies the type of food and related ingredients contained in the photo. For example, if the posted photo is of soup, the name of the dish, such as "minestrone" or "potage," will be identified, and vegetables and spices will be recognized as ingredients.

[0039] Subsequently, the server uses the identified ingredient information to query a nutrition database or an external nutrition information API to retrieve the nutritional components of each ingredient. This allows the server to calculate the overall nutritional balance of the user's past meals and identify any nutritional deficiencies.

[0040] Based on the above analysis, the server selects suggested dishes to supplement any missing nutrients. This may include menus and recipes from restaurants in the user's area. The selected dishes and ingredients will help improve the user's nutritional balance.

[0041] Finally, the server notifies the user of the selected suggestions on their device. Users can review the suggestions through the app or web interface on their device and use them to plan their next meal. The suggestions also include information on nearby partner restaurants, giving users more options to actually try the suggested dishes.

[0042] This invention will be implemented in a way that allows users to easily manage their daily dietary health and efficiently compensate for deficiencies in specific nutrients.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] Users link their social media accounts through the system's interface. This allows the server to gain access to the user's social media posts.

[0046] Step 2:

[0047] The server uses SNS APIs to periodically or at specified intervals retrieve image data related to cooking from user accounts. The data retrieved from SNS includes the image URL and associated text information.

[0048] Step 3:

[0049] The server runs an image recognition AI to analyze the acquired food images. Through this analysis, it identifies the type of dish and its ingredients, and generates the dish name and a list of ingredients.

[0050] Step 4:

[0051] The server uses the identified material information to access a nutrition database or an external nutrition information API to retrieve nutritional component data for each material.

[0052] Step 5:

[0053] Based on the acquired nutritional data, the server analyzes the nutritional balance calculated from the user's past meals, identifying necessary nutrients and nutrients in excess.

[0054] Step 6:

[0055] The server selects dishes to supplement any missing nutrients. This selection process takes into account menus from partner restaurants and general recipe information.

[0056] Step 7:

[0057] The server selects a proposal and prepares to notify the user's device and display it. The notification is sent via in-app messages or push notifications.

[0058] Step 8:

[0059] The device provides the user with an interface to review suggested dishes and ingredients, and the user receives and views information to help them choose their next meal.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In recent years, healthy nutritional management has become increasingly important in individual diets, but it is difficult for individuals to accurately grasp the nutrients in the foods they consume daily and plan a balanced diet. Furthermore, the lack of nutritional information about food when eating out or during cooking makes it difficult for individuals to determine how to improve their balance, posing an obstacle to maintaining a healthy lifestyle.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for acquiring linked information via the user's information terminal, means for recognizing food using image processing technology based on the acquired information, and means for evaluating nutrients based on the recognized food information. This enables the automatic analysis of nutritional information of foods consumed daily by an individual and, based on the results, makes it possible to suggest meals that take nutritional balance into consideration.

[0065] A "user information terminal" refers to a device such as a mobile phone or computer used by a user, which is used to acquire information in conjunction with a system.

[0066] "Image processing technology" refers to the techniques used to analyze images represented as digital data and to understand and recognize their content.

[0067] "Means of recognizing food" refers to processes or systems for identifying food contained in acquired images and determining its type and attributes.

[0068] "Nutrient evaluation" is a method for analyzing the nutritional components of a specific food in detail and examining its nutritional value and impact on health.

[0069] "Food information" refers to detailed data related to a particular food product, such as its nutritional components, ingredients, and cooking methods.

[0070] "Nutritional balance" is an indicator that shows whether the various nutrients necessary for maintaining health are being consumed appropriately.

[0071] This invention is a system designed to support users in improving their eating habits. It utilizes images of food that users post on social media on a daily basis and automatically analyzes their nutritional information.

[0072] Users can begin analyzing food images by linking their SNS accounts with this system via their personal information terminal. Through this linkage, the system retrieves user-submitted data via the SNS API and sends the images to the server.

[0073] The server analyzes the collected images using image recognition modules such as Python's TENSORFLOW® and PyTorch. This analysis process utilizes image processing techniques to identify the type of food and its ingredients. For example, if an image contains "pasta," the system will recognize ingredients such as "tomatoes" and "olive oil."

[0074] Next, the server sends queries to external nutrition databases and APIs to proceed with the process of obtaining nutritional information for the identified ingredients. Based on the nutritional data obtained from these databases, the system compares it with the user's past dietary records to evaluate nutritional balance. For example, if the system determines that the user is particularly deficient in "calcium," it will generate meal suggestions to compensate for that deficiency.

[0075] The suggested meals are generated by a generative AI model and associated with local restaurant information based on the user's location. This information is sent from the server to the user's device, and the user can view it through an application or web interface.

[0076] An example of a specific prompt is, "Analyze the food images posted by the user on social media, identify the nutritional information, and generate healthy meal suggestions." This prompt serves as an instruction to the generating AI model and is used to provide the user with suggestions based on the analysis results.

[0077] This system makes it easy for users to adopt a health-conscious diet and effectively supplement their nutritional needs.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] Users link their social networking service (SNS) accounts to the system using an information terminal. The input required is the authentication information for the user's SNS account. Users grant access to the system and set the necessary API permissions through a dedicated application or web interface. This allows the server to access the user's public posts.

[0081] Step 2:

[0082] The server collects user-submitted image data using the SNS API. The input is the URL of the user's posted image. The server periodically sends requests to the API to check for the latest posts. This data is stored on the server as images of food.

[0083] Step 3:

[0084] The server inputs the collected image data into an image recognition module for analysis to identify food items. The input consists of stored image data of dishes. The server uses TensorFlow or PyTorch to identify the food items contained in the images. The output is a list of food names and ingredients.

[0085] Step 4:

[0086] The server queries nutrition databases and external APIs for information on the ingredients of identified foods and retrieves the relevant nutritional components. The input is a list of food names and ingredients. The server accesses the database and retrieves the nutrients for each ingredient as numerical values. The output is the nutritional information for each ingredient.

[0087] Step 5:

[0088] The server evaluates the user's nutritional balance based on acquired nutritional information. Inputs include nutritional information and the user's past meal history. The server identifies deficient nutrients and generates meal suggestions based on the results. Output is the evaluation of the user's nutritional balance.

[0089] Step 6:

[0090] The server uses a generative AI model to create meal suggestions for improving nutritional balance. The input is the user's nutritional balance assessment result. The server considers the user's geographical information and generates suggestions that combine locally appropriate meals and information on affiliated restaurants. The output is the suggested meal plan.

[0091] Step 7:

[0092] The server notifies the user's information terminal of the meal suggestions it has created. The input is the suggested meal plan. The user receives push notifications and alerts using an application on their terminal and checks the suggested content. This allows the user to put their healthy meal plan into action.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] In modern life, people are required to efficiently manage their health amidst busy daily routines. However, it is difficult for the average consumer to analyze their daily diet in detail and accurately supplement the necessary nutrients. Furthermore, with the increasing use of eating out and delivery services, there is a lack of accurate information to help people choose healthy menus. Therefore, there is a need for a system that can automatically analyze nutritional information from food images posted on social media, and then provide healthy meal suggestions and immediately available cooking options based on the results.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes a mechanism for receiving posted information, a recognition mechanism for extracting cooking targets from image information, a mechanism for analyzing nutritional status based on the extracted cooking targets, a mechanism for presenting options for providing selected cooking targets in cooperation with nearby distribution services, and a mechanism for notifying the user of the recommended cooking targets. This enables users to accurately understand their nutritional status based on images obtained from social media and to receive healthy meals accordingly.

[0098] "Posted information" refers to information, including images and text, that users make public through social networking services, etc.

[0099] "Image information" refers to digital data that visually represents food or other objects.

[0100] "Items to be cooked" refers to individual ingredients or menu items that are served as a dish.

[0101] A "recognition mechanism" is a computer program or algorithm used to analyze image information and identify the food being cooked.

[0102] "Nutritional status" refers to the analysis results that show the types and amounts of nutrients contained in the food being cooked.

[0103] A "delivery service" refers to a delivery system or operator that provides food or ingredients to customers.

[0104] A "mechanism for presenting options" is a platform that provides users with available delivery menus and restaurant information based on analysis results.

[0105] A "notification mechanism" refers to a digital communication method or application used to notify users of information regarding proposed cooking targets and health improvements.

[0106] The system that realizes this invention consists of a server, a user's terminal, and an external API. First, an application installed on the user's terminal links with the SNS account. Through this link, the server can periodically retrieve the user's posted information via the SNS API.

[0107] The server uses a recognition mechanism to identify the food being cooked from image information in order to analyze the acquired post information. This recognition uses advanced image analysis software, TensorFlow and OpenCV, to identify specific dishes and ingredients. Subsequently, to analyze the nutritional status of the identified food, nutritional information is obtained from sources such as the USDA FoodData Central API, and a comprehensive nutritional analysis is performed.

[0108] Based on the analysis results, the server suggests healthy meals. This process involves collaborating with nearby delivery services to provide options based on the user's location. Delivery service APIs such as DoorDash and UberEats are used to present these options. Users can review the suggested options through their device's interface and place orders as needed.

[0109] As a concrete example, a Caesar salad might be identified from an image a user posted on social media, and based on an analysis of their past nutritional balance, the system might suggest adding grilled chicken to supplement their protein intake. The user can then easily order from a nearby restaurant based on this information.

[0110] An example of a prompt for a generative AI model is, "Please analyze the nutritional content of food images posted on social media and suggest healthy delivery menus."

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The device links the SNS account with the application. When the user logs into the app and performs SNS authentication, the app and the SNS API are linked. The input requires the user's authentication information, and the output is permission to retrieve the user's posted information from the SNS.

[0114] Step 2:

[0115] The device uses the SNS API to send user-submitted images to the server. The server periodically collects submission information, particularly image data. The input in this step is image data, and the output is saving it to the image database on the server.

[0116] Step 3:

[0117] The server passes image data to a recognition mechanism to identify the food being cooked. TensorFlow and OpenCV are used for image recognition processing to analyze ingredients and dish names from the image. The input here is image data, and the output is a list of dish names and ingredients. Specifically, this involves AI-based image recognition.

[0118] Step 4:

[0119] The server retrieves nutritional information based on identified dishes and ingredients. It uses the USDA FoodData Central API to obtain nutritional data related to the dishes and analyze their nutritional status. Inputs include dish names and ingredient information, while output is a nutritional information table. Data processing includes sending queries to the API and analyzing the responses.

[0120] Step 5:

[0121] The server suggests healthy meals based on the analyzed nutritional information. It uses an AI model to generate options to ensure nutritional balance. The input is nutritional information, and the output is a list of suggested healthy menus. This includes generating prompts and providing them to the AI.

[0122] Step 6:

[0123] The server retrieves information about restaurants that offer the suggested menu using the user's location and a delivery service API. The input is the user's location and the suggested menu, and the output is a list of available restaurants. This involves specific API calls and response processing.

[0124] Step 7:

[0125] The terminal notifies the user of healthy meal suggestions and store information received from the server. The user can review and select from the suggested information. The input is the suggested information and store information, and the output is the notification presented to the user. Specific sending actions are included.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention is a system that analyzes food images posted on social media to evaluate a user's nutritional balance, suggests appropriate meals, and recognizes the user's emotions, reflecting them in the meal suggestions. This system is implemented as follows.

[0128] First, users link their social media accounts with this system, allowing the server to access their posts. The server uses the social media API to periodically retrieve the relevant food image data, or upon user request. The retrieved image data also includes the text information of the posts.

[0129] Next, the server uses image recognition AI to analyze the acquired food images. This analysis extracts the identified dish name and related ingredient information. Based on this, the server sends requests to a nutrition database or external nutrition information API to retrieve relevant nutritional components and calculate the user's nutritional balance.

[0130] The server then uses an emotion engine to evaluate the user's emotions based on the content and images of their posts. For example, if a user posts a photo with a positive comment, the emotion engine might recognize that emotion as "joy."

[0131] The server selects dishes to suggest based on calculated nutritional balance and perceived emotions. For users with positive emotions, it can suggest dishes that help maintain that mood, while for users with negative emotions, it can suggest dishes that help improve their mood.

[0132] Finally, the server notifies the user's device of the selected recommendations. The device displays the recommended dishes and the reasons for their selection, which the user can use to help with their next meal choice. The recommendations may also include information on nearby partner restaurants based on the user's location.

[0133] For example, if a user posts a photo of a summer barbecue with the comment "It was a fun day!", the server analyzes the nutritional information derived from the ingredients in the image and recognizes the emotion of "joy." Based on this, it suggests meals that promote further positive experiences, such as nutritious salads and fresh juices. In this way, by making meal suggestions that take the user's mood into consideration, more personalized health promotion becomes possible.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] Users link their social media accounts to this system. This grants the system access to their social media posts.

[0137] Step 2:

[0138] The server uses the SNS API to retrieve the user's latest posts. These posts include images of food uploaded by the user and the accompanying text.

[0139] Step 3:

[0140] The server uses an image processing module to analyze the acquired images and identify the type of dish and its main ingredients. AI technology is used to pinpoint the exact dish name and ingredients.

[0141] Step 4:

[0142] The server activates an emotion engine to analyze the user's emotional state from the content of the posted text and images. This analysis uses natural language processing techniques to recognize emotional expressions in the text.

[0143] Step 5:

[0144] The server uses the ingredient information of the identified dish to query a nutritional information database and retrieve nutritional data for each component. Based on this, it calculates the user's nutritional balance and analyzes any excesses or deficiencies.

[0145] Step 6:

[0146] The server selects a suitable meal for the user based on calculated nutritional balance and analyzed emotional state. The meal selection takes into account the user's mood and ensures that the necessary nutrients are provided.

[0147] Step 7:

[0148] The server selects a dish and notifies the user's device. The notification can include the reason for the suggestion, the nutritional characteristics of the dish, and its emotional impact.

[0149] Step 8:

[0150] The device provides the user with an interface that displays suggested dishes and their details. The user considers the suggestions and uses them as a reference when deciding what to eat next.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] Maintaining a nutritionally balanced diet is crucial for promoting health in modern life. However, providing nutritional information and meal suggestions tailored to each user's individual dietary habits and emotional state is challenging, and an efficient method is needed. Especially with the increasing number of users posting about their meals on social media, analyzing this information in real time and providing appropriate suggestions could contribute to users' ongoing health management.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes a device for receiving user submissions, an identification device for identifying food from image data contained in the received submissions, a device for calculating nutritional composition based on the identified food, a device for selecting food to suggest, a device for notifying the user of the suggested food, and a device for analyzing the user's emotions and reflecting them in the suggestions. This makes it possible to provide meal suggestions that simultaneously consider the user's nutritional status and emotional state.

[0156] A "device for receiving user posts" refers to hardware or software used to acquire digital data shared by users on social networking services (SNS) or online platforms.

[0157] A "food identification device from image data" is a device that uses image recognition technology to identify food or dishes shown in acquired image data.

[0158] A "device for calculating nutritional composition based on identified foods" is a device that quantifies the nutrients of identified foods and evaluates their balance based on the ingredients and component information of those foods.

[0159] A "device for selecting suggested foods" is a device that combines calculated nutritional information with emotional analysis data to select foods and dishes suitable for the user.

[0160] A "device that notifies users of proposed food items" is a device that has a system function for transmitting information about selected food items or dishes to the user's device.

[0161] A "device that analyzes user emotions and reflects them in suggestions" is a device that reads emotions from text and images and uses the identified emotions to provide personalized food suggestions.

[0162] This invention is implemented through a system that analyzes users' food posts and emotions, and makes suggestions based on the results. Users link their social media accounts to the system, allowing the server to receive data from these posts.

[0163] The server uses SNS APIs to periodically retrieve posted data, including text and images. For image data, it utilizes image recognition AI such as TensorFlow and PyTorch to identify food items and dishes. Based on this, it extracts information about the food's ingredients and then uses nutrition databases and nutrition information APIs to obtain and calculate the nutritional components of the food item.

[0164] Sentiment analysis utilizes automated text processing. Specifically, natural language processing libraries (such as NLTK or SpaCy) are used to analyze the sentiment from submitted text and evaluate whether the sentiment is positive or negative.

[0165] After this, the server will provide the user with optimal meal suggestions based on calculated nutritional content and emotional analysis. The selected suggestions will include meals that take key nutrients into consideration and foods that match the user's emotions. The user's device will be notified of meal or cooking suggestions, and these suggestions may include information on nearby partner stores based on location data.

[0166] For example, if a user posts an image to social media with the comment "A fun day of barbecuing," the server uses AI to analyze the image and identify the emotion as "joy." Based on this information, it becomes possible to suggest meals that promote a more positive experience, such as nutritious salads and fruit juices.

[0167] An example of a prompt to input into the generative AI model would be, "Please suggest a meal when a user posts an outdoor image with the comment 'It was a fun day!'" In this way, the goal is to provide a personalized experience that is tailored to the user's actions and emotions.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] Users link their social media accounts to the system, allowing the server to access their posts. The input information consists of the user's account authentication credentials. This prepares the server to retrieve post data using the API. Specifically, authentication protocols such as OAuth are used to securely configure access rights.

[0171] Step 2:

[0172] The server receives image and text data posted by users via the SNS API. The input consists of newly posted images and text. As output, this data is stored within the system and used in the next analysis step. Specifically, the API call pulls the data and saves it to the server's storage.

[0173] Step 3:

[0174] The server utilizes image recognition AI to analyze the acquired image data. The input is the image data acquired in step 2. This analysis identifies food and dishes within the image and extracts their information. The output is the identified food name and ingredient information. Specifically, it runs an image classification model using TensorFlow or PyTorch to estimate the type and ingredients of the dish.

[0175] Step 4:

[0176] The server retrieves nutritional information by accessing nutrition databases and external APIs based on the food name and ingredient information. The input is the food information identified in step 3. The output is detailed nutritional component data for each food. Specifically, it executes queries to external databases to retrieve nutritional components related to the food.

[0177] Step 5:

[0178] The server uses natural language processing techniques to analyze user sentiment from posted text. The input is the text data obtained in step 2. The output is the analyzed sentiment information (e.g., positive, negative). Specifically, it uses NLTK or SpaCy to analyze the text and calculate the sentiment score.

[0179] Step 6:

[0180] The server selects dishes to suggest to the user based on nutritional and emotional information. Inputs are nutritional data and emotional information. Output is a list of recommended dishes. Specifically, it executes an algorithm that selects the optimal dish based on the user's nutritional status and emotional state.

[0181] Step 7:

[0182] The server sends the selected recipe suggestions to the user's device. The input is the list of dishes from step 6. The output is the suggestions received by the user. Specifically, a push notification is sent via a dedicated app or similar, displaying the suggested content on the user's screen.

[0183] Step 8:

[0184] The terminal displays the submitted suggestions to the user, providing information to help them make their next meal selection. Input is a suggestion notification from the server. Output is visual feedback to the user. Specifically, it displays dish details and selection reasons on a GUI, and provides additional information and links if necessary.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] In modern food services, providing appropriate meals based on individual dietary habits and health conditions remains challenging. In particular, there is a need for technology that effectively suggests meals that consider the user's nutritional balance and emotional state, and delivers them quickly. Furthermore, coordinating with external supply services to achieve a seamless ordering experience is also a challenge.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for receiving user posts, identification means for identifying meals from images contained in the received posts, and means for calculating nutritional balance based on the identified meals. This enables seamless ordering of meal suggestions and delivery services tailored to the individual user's nutritional needs and preferences.

[0190] A "user" refers to an individual who receives meal suggestions using this system.

[0191] "Posting" refers to information, especially images and text, that a user uploads to an online information sharing service.

[0192] "Images" refer to the visual data included in a post and are a source of information for identifying the food.

[0193] "Meal" refers to food or dishes identified by image recognition technology.

[0194] "Nutritional balance" refers to the proportion of nutrients contained in a specific meal and is a standard used to evaluate a user's health status.

[0195] "Emotions" refer to the psychological state perceived from the user's posted content and images, and are factors that influence meal suggestions.

[0196] "Recommended meals" refer to meals recommended to users based on calculated nutritional balance and emotional state.

[0197] "External supply services" refer to third-party distributors or sellers who actually provide the proposed meals to users.

[0198] "Location information" refers to data that indicates the user's current location and is used to select suppliers that provide the suggested meals.

[0199] The system used to implement this application primarily consists of a server, a user terminal, and integration with external services. The server first receives SNS posts from the user's terminal. These posts contain images and text related to food.

[0200] The server analyzes food images acquired using image recognition technology to identify specific meals. This may involve using image recognition services such as Google® Cloud Vision API. Based on the meal identified from the image, nutritional data is retrieved from an external nutrition information database or API. For example, the Nutritionix API could be used.

[0201] Furthermore, the server analyzes the text within the posts to determine the user's emotions. This analysis can utilize an emotion recognition engine such as IBM Watson® Tone Analyzer. Based on the analysis results, it generates meal suggestions that take into account specific nutritional balances and emotional states.

[0202] Next, the server not only notifies the user of the suggested meal but also allows the user to order the suggested meal through an external supply service. It is possible to process orders by utilizing APIs of delivery services such as the Uber Eats API.

[0203] For example, if a user posts a photo of pancakes with the comment, "Sunday brunch was amazing!", the system might suggest a salad that is more nutritious compared to that snack. The suggested meal would then be presented to the user along with relevant nutritional information, and they could then order it.

[0204] An example of a prompt message might be: "Extract the name of the dish and the sentiment from the user's social media post, and suggest an appropriate meal based on that information. Check if the suggested dish is available for order through an existing delivery service."

[0205] Thus, this embodiment of the invention involves a server integrating and utilizing image recognition, emotion recognition, and nutritional analysis technologies to provide customized meal suggestions and ordering functions to individual users.

[0206] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0207] Step 1:

[0208] The server receives user posts via the SNS API. The input consists of user posts, which include image data and text information. This allows the server to import the received post data.

[0209] Step 2:

[0210] The server uses an image recognition engine to analyze the received image data. The input is the image data acquired in step 1, and the output is the identified meal or dish name. Here, the dish is identified from the image, and information is extracted for reference in the database.

[0211] Step 3:

[0212] The server sends a request to an external nutrition information API to retrieve nutritional data based on the identified meal. The input is the name of the dish, which is the output of step 2, and the output is the nutritional information associated with that dish. This provides the data necessary to calculate nutritional balance.

[0213] Step 4:

[0214] The server uses an emotion analysis engine to evaluate the user's emotions from the text information of the post. The input is the text information obtained in step 1, and the output is data indicating the user's emotional state. This clarifies the emotional indicators that should be considered when making meal suggestions.

[0215] Step 5:

[0216] The server combines nutritional balance data and emotional data to generate optimal meal suggestions for the user. The input is the output data from steps 3 and 4, and the output is the suggested meals. Calculations and selections are then performed to determine the appropriate meal.

[0217] Step 6:

[0218] The server notifies the user terminal of the suggested meal and confirms the possibility of ordering from an external supply service. The input is the suggested meal obtained in step 5, and the output is the notification to the user and ordering options. The user then receives the meal suggestion and can place an order immediately.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention is a system that automatically analyzes nutritional information from food images posted by users on social media and suggests healthy meals. This system is implemented in the following specific form.

[0236] First, users link their social media accounts with this system. This allows the server to efficiently retrieve food photos that have been publicly shared from the user's account. The server periodically or in response to user actions uses social media APIs to collect posted image data.

[0237] Next, the server uses an AI module equipped with image recognition technology to analyze the acquired images. The analysis process identifies the type of food and related ingredients contained in the photo. For example, if the posted photo is of soup, the name of the dish, such as "minestrone" or "potage," will be identified, and vegetables and spices will be recognized as ingredients.

[0238] Subsequently, the server sends queries to a nutrition database or an external nutrition information API based on the identified ingredient information to retrieve the nutritional components of each ingredient. This allows the nutritional balance of the user's past meals to be calculated, and any deficient nutrients to be identified.

[0239] Based on the above analysis, the server selects dishes to suggest that supplement any missing nutrients. This may include menus and recipes from restaurants in the user's area. The selected dishes and ingredients will help improve the user's nutritional balance.

[0240] Finally, the server notifies the user of the selected suggestions on their device. Users can review the suggestions through the app or web interface on their device and use them to plan their next meal. The suggestions also include information on nearby partner restaurants, giving users more options to actually try the suggested dishes.

[0241] This invention will be implemented in a way that allows users to easily manage their daily dietary health and efficiently compensate for deficiencies in specific nutrients.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] Users link their social media accounts through the system's interface. This allows the server to gain access to the user's social media posts.

[0245] Step 2:

[0246] The server uses SNS APIs to periodically or at specified intervals retrieve image data related to cooking from user accounts. The data retrieved from SNS includes the image URL and associated text information.

[0247] Step 3:

[0248] The server runs an image recognition AI to analyze the acquired food images. Through this analysis, it identifies the type of dish and its ingredients, and generates the dish name and a list of ingredients.

[0249] Step 4:

[0250] The server uses the identified material information to access a nutrition database or an external nutrition information API to retrieve nutritional component data for each material.

[0251] Step 5:

[0252] Based on the acquired nutritional data, the server analyzes the nutritional balance calculated from the user's past meals, identifying necessary nutrients and nutrients in excess.

[0253] Step 6:

[0254] The server selects dishes to supplement any missing nutrients. This selection process takes into account menus from partner restaurants and general recipe information.

[0255] Step 7:

[0256] The server selects a proposal and prepares to notify the user's device and display it. The notification is sent via in-app messages or push notifications.

[0257] Step 8:

[0258] The device provides the user with an interface to review suggested dishes and ingredients, and the user receives and views information to help them choose their next meal.

[0259] (Example 1)

[0260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0261] In recent years, healthy nutritional management has become increasingly important in individual diets, but it is difficult for individuals to accurately grasp the nutrients in the foods they consume daily and plan a balanced diet. Furthermore, the lack of nutritional information about food when eating out or during cooking makes it difficult for individuals to determine how to improve their balance, posing an obstacle to maintaining a healthy lifestyle.

[0262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0263] In this invention, the server includes means for acquiring linked information via the user's information terminal, means for recognizing food using image processing technology based on the acquired information, and means for evaluating nutrients based on the recognized food information. This enables the automatic analysis of nutritional information of foods consumed daily by an individual and, based on the results, makes it possible to suggest meals that take nutritional balance into consideration.

[0264] A "user information terminal" refers to a device such as a mobile phone or computer used by a user, which is used to acquire information in conjunction with a system.

[0265] "Image processing technology" refers to the techniques used to analyze images represented as digital data and to understand and recognize their content.

[0266] "Means of recognizing food" refers to processes or systems for identifying food contained in acquired images and determining its type and attributes.

[0267] "Nutrient evaluation" is a method for analyzing the nutritional components of a specific food in detail and examining its nutritional value and impact on health.

[0268] "Food information" refers to detailed data related to a particular food product, such as its nutritional components, ingredients, and cooking methods.

[0269] "Nutritional balance" is an indicator that shows whether the various nutrients necessary for maintaining health are being consumed appropriately.

[0270] This invention is a system designed to support users in improving their eating habits. It utilizes images of food that users post on social media on a daily basis and automatically analyzes their nutritional information.

[0271] Users can begin analyzing food images by linking their SNS accounts with this system via their personal information terminal. Through this linkage, the system retrieves user-submitted data via the SNS API and sends the images to the server.

[0272] The server analyzes the collected images using image recognition modules such as Python's TensorFlow and PyTorch. This analysis process utilizes image processing techniques to identify the type of food and its ingredients. For example, if an image contains "pasta," the system will recognize ingredients such as "tomatoes" and "olive oil."

[0273] Next, the server sends queries to external nutrition databases and APIs to proceed with the process of obtaining nutritional information for the identified ingredients. Based on the nutritional data obtained from these databases, the system compares it with the user's past dietary records to evaluate nutritional balance. For example, if the system determines that the user is particularly deficient in "calcium," it will generate meal suggestions to compensate for that deficiency.

[0274] The suggested meals are generated by a generative AI model and associated with local restaurant information based on the user's location. This information is sent from the server to the user's device, and the user can view it through an application or web interface.

[0275] An example of a specific prompt is, "Analyze the food images posted by the user on social media, identify the nutritional information, and generate healthy meal suggestions." This prompt serves as an instruction to the generating AI model and is used to provide the user with suggestions based on the analysis results.

[0276] This system makes it easy for users to adopt a health-conscious diet and effectively supplement their nutritional needs.

[0277] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0278] Step 1:

[0279] Users link their social networking service (SNS) accounts to the system using an information terminal. The input required is the authentication information for the user's SNS account. Users grant access to the system and set the necessary API permissions through a dedicated application or web interface. This allows the server to access the user's public posts.

[0280] Step 2:

[0281] The server collects the user's posted image data using the SNS API. The input is the URL of the user's posted image. The server periodically sends requests to the API to check for the latest posts. This data is saved on the server as images of dishes.

[0282] Step 3:

[0283] The server inputs the collected image data into an image recognition module for analysis to identify food. The input is the saved dish image data. The server uses TensorFlow or PyTorch to identify the food contained in the image. The output is a list of food names and ingredients.

[0284] Step 4:

[0285] The server queries the nutritional database or external API with the ingredient information of the identified food to obtain relevant nutritional components. The input is the list of food names and ingredients. The server accesses the database to obtain the nutrients of each ingredient as numerical values. The output is the nutritional component information of each ingredient.

[0286] Step 5:

[0287] The server evaluates the user's nutritional balance based on the obtained nutritional components. The input is the nutritional component information and the user's past diet history. The server identifies the lacking nutrients and generates diet proposals based on the results. The output is the evaluation result of the user's nutritional balance.

[0288] Step 6:

[0289] The server uses a generation AI model to create diet proposals for improving the nutritional balance. The input is the evaluation result of the user's nutritional balance. The server considers the user's geographical information and generates proposals that combine region - suitable diets and partner store information. The output is the proposed diet plan.

[0290] Step 7:

[0291] The server notifies the user's information terminal of the meal suggestions it has created. The input is the suggested meal plan. The user receives push notifications and alerts using an application on their terminal to confirm the suggestions. This allows the user to put their healthy meal plan into action.

[0292] (Application Example 1)

[0293] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0294] In modern life, people are required to efficiently manage their health amidst busy daily routines. However, it is difficult for the average consumer to analyze their daily diet in detail and accurately supplement the necessary nutrients. Furthermore, with the increasing use of eating out and delivery services, there is a lack of accurate information to help people choose healthy menus. Therefore, there is a need for a system that can automatically analyze nutritional information from food images posted on social media, and then provide healthy meal suggestions and immediately available cooking options based on the results.

[0295] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0296] In this invention, the server includes a mechanism for receiving posted information, a recognition mechanism for extracting cooking targets from image information, a mechanism for analyzing nutritional status based on the extracted cooking targets, a mechanism for presenting options for providing selected cooking targets in cooperation with nearby distribution services, and a mechanism for notifying the user of the recommended cooking targets. This enables users to accurately understand their nutritional status based on images obtained from social media and to receive healthy meals accordingly.

[0297] "Posted information" refers to information, including images and text, that users make public through social networking services, etc.

[0298] "Image information" refers to digital data that visually represents food or other objects.

[0299] "Items to be cooked" refers to individual ingredients or menu items that are served as a dish.

[0300] A "recognition mechanism" is a computer program or algorithm used to analyze image information and identify the food being cooked.

[0301] "Nutritional status" refers to the analysis results that show the types and amounts of nutrients contained in the food being cooked.

[0302] A "delivery service" refers to a delivery system or operator that provides food or ingredients to customers.

[0303] A "mechanism for presenting options" is a platform that provides users with available delivery menus and restaurant information based on analysis results.

[0304] A "notification mechanism" refers to a digital communication method or application used to notify users of information regarding proposed cooking targets and health improvements.

[0305] The system that realizes this invention consists of a server, a user's terminal, and an external API. First, an application installed on the user's terminal links with the SNS account. Through this link, the server can periodically retrieve the user's posted information via the SNS API.

[0306] The server uses a recognition mechanism to identify the cooking target from the image information in order to analyze the acquired post information. For this recognition, advanced image analysis software such as TensorFlow and OpenCV is used, and specific dishes and ingredients are identified thereby. Thereafter, in order to analyze the nutritional status of the identified cooking target, nutritional component information is acquired from the USDA FoodData Central API or the like, and comprehensive nutritional analysis is performed.

[0307] Based on the analysis results, the server makes proposals for healthy diets. In this process, it cooperates with neighboring delivery services and provides options based on the user's location information. For presenting the options, delivery service APIs such as DoorDash and UberEats are utilized as examples. The user can confirm the proposed options through the interface of the terminal and place an order if necessary.

[0308] As a specific example, "Caesar salad" is identified from the image posted by the user on SNS, and based on the analysis of the past nutritional balance, the addition of "grilled chicken" is proposed for protein supplementation. The user can easily place an order from nearby restaurants based on this information.

[0309] As an example of a prompt sentence for the generative AI model, there is "Analyze the nutritional components from the cooking image posted on SNS and tell me the healthy delivery menus that can be proposed".

[0310] The flow of the specific processing in Application Example 1 will be described using FIG. 12.

[0311] Step 1:

[0312] The terminal links the SNS account and the application. When the user logs in to the application and performs SNS authentication, the APIs of the application and SNS are linked. As input, the user's authentication information is required, and as output, the right to acquire the user's post information from SNS is obtained.

[0313] Step 2:

[0314] The device uses the SNS API to send user-submitted images to the server. The server periodically collects submission information, particularly image data. The input in this step is image data, and the output is saving it to the image database on the server.

[0315] Step 3:

[0316] The server passes image data to a recognition mechanism to identify the food being cooked. TensorFlow and OpenCV are used for image recognition processing to analyze ingredients and dish names from the image. The input here is image data, and the output is a list of dish names and ingredients. Specifically, this involves AI-based image recognition.

[0317] Step 4:

[0318] The server retrieves nutritional information based on identified dishes and ingredients. It uses the USDA FoodData Central API to obtain nutritional data related to the dishes and analyze their nutritional status. Inputs include dish names and ingredient information, while output is a nutritional information table. Data processing includes sending queries to the API and analyzing the responses.

[0319] Step 5:

[0320] The server suggests healthy meals based on the analyzed nutritional information. It uses an AI model to generate options to ensure nutritional balance. The input is nutritional information, and the output is a list of suggested healthy menus. This includes generating prompts and providing them to the AI.

[0321] Step 6:

[0322] The server retrieves information about restaurants that offer the suggested menu using the user's location and a delivery service API. The input is the user's location and the suggested menu, and the output is a list of available restaurants. This involves specific API calls and response processing.

[0323] Step 7:

[0324] The terminal notifies the user of healthy meal suggestions and store information received from the server. The user can review and select from the suggested information. The input is the suggested information and store information, and the output is the notification presented to the user. Specific sending actions are included.

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] This invention is a system that analyzes food images posted on social media to evaluate a user's nutritional balance, suggests appropriate meals, and recognizes the user's emotions, reflecting them in the meal suggestions. This system is implemented as follows.

[0327] First, users link their social media accounts with this system, allowing the server to access their posts. The server uses the social media API to periodically retrieve the relevant food image data, or upon user request. The retrieved image data also includes the text information of the posts.

[0328] Next, the server uses image recognition AI to analyze the acquired food images. This analysis extracts the identified dish name and related ingredient information. Based on this, the server sends requests to a nutrition database or external nutrition information API to retrieve relevant nutritional components and calculate the user's nutritional balance.

[0329] The server then uses an emotion engine to evaluate the user's emotions based on the content and images of their posts. For example, if a user posts a photo with a positive comment, the emotion engine might recognize that emotion as "joy."

[0330] The server selects dishes to suggest based on calculated nutritional balance and perceived emotions. For users with positive emotions, it can suggest dishes that help maintain that mood, while for users with negative emotions, it can suggest dishes that help improve their mood.

[0331] Finally, the server notifies the user's device of the selected recommendations. The device displays the recommended dishes and the reasons for their selection, which the user can use to help with their next meal choice. The recommendations may also include information on nearby partner restaurants based on the user's location.

[0332] For example, if a user posts a photo of a summer barbecue with the comment "It was a fun day!", the server analyzes the nutritional information derived from the ingredients in the image and recognizes the emotion of "joy." Based on this, it suggests meals that promote further positive experiences, such as nutritious salads and fresh juices. In this way, by making meal suggestions that take the user's mood into consideration, more personalized health promotion becomes possible.

[0333] The following describes the processing flow.

[0334] Step 1:

[0335] Users link their social media accounts to this system. This grants the system access to their social media posts.

[0336] Step 2:

[0337] The server uses the SNS API to retrieve the user's latest posts. These posts include images of food uploaded by the user and the accompanying text.

[0338] Step 3:

[0339] The server uses an image processing module to analyze the acquired images and identify the type of dish and its main ingredients. AI technology is used to pinpoint the exact dish name and ingredients.

[0340] Step 4:

[0341] The server activates an emotion engine to analyze the user's emotional state from the content of the posted text and images. This analysis uses natural language processing techniques to recognize emotional expressions in the text.

[0342] Step 5:

[0343] The server uses the ingredient information of the identified dish to query a nutritional information database and retrieve nutritional data for each component. Based on this, it calculates the user's nutritional balance and analyzes any excesses or deficiencies.

[0344] Step 6:

[0345] The server selects a suitable meal for the user based on calculated nutritional balance and analyzed emotional state. The meal selection takes into account the user's mood and ensures that the necessary nutrients are provided.

[0346] Step 7:

[0347] The server selects a dish and notifies the user's device. The notification can include the reason for the suggestion, the nutritional characteristics of the dish, and its emotional impact.

[0348] Step 8:

[0349] The device provides the user with an interface that displays suggested dishes and their details. The user considers the suggestions and uses them as a reference when deciding what to eat next.

[0350] (Example 2)

[0351] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0352] Maintaining a nutritionally balanced diet is crucial for promoting health in modern life. However, providing nutritional information and meal suggestions tailored to each user's individual dietary habits and emotional state is challenging, and an efficient method is needed. Especially with the increasing number of users posting about their meals on social media, analyzing this information in real time and providing appropriate suggestions could contribute to users' ongoing health management.

[0353] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0354] In this invention, the server includes a device for receiving user submissions, an identification device for identifying food from image data contained in the received submissions, a device for calculating nutritional composition based on the identified food, a device for selecting food to suggest, a device for notifying the user of the suggested food, and a device for analyzing the user's emotions and reflecting them in the suggestions. This makes it possible to provide meal suggestions that simultaneously consider the user's nutritional status and emotional state.

[0355] A "device for receiving user posts" refers to hardware or software used to acquire digital data shared by users on social networking services (SNS) or online platforms.

[0356] A "food identification device from image data" is a device that uses image recognition technology to identify food or dishes shown in acquired image data.

[0357] A "device for calculating nutritional composition based on identified foods" is a device that quantifies the nutrients of identified foods and evaluates their balance based on the ingredients and component information of those foods.

[0358] A "device for selecting suggested foods" is a device that combines calculated nutritional information with emotional analysis data to select foods and dishes suitable for the user.

[0359] A "device that notifies users of proposed food items" is a device that has a system function for transmitting information about selected food items or dishes to the user's device.

[0360] A "device that analyzes user emotions and reflects them in suggestions" is a device that reads emotions from text and images and uses the identified emotions to provide personalized food suggestions.

[0361] This invention is implemented through a system that analyzes users' food posts and emotions, and makes suggestions based on the results. Users link their social media accounts to the system, allowing the server to receive data from these posts.

[0362] The server uses SNS APIs to periodically retrieve posted data, including text and images. For image data, it utilizes image recognition AI such as TensorFlow and PyTorch to identify food items and dishes. Based on this, it extracts information about the food's ingredients and then uses nutrition databases and nutrition information APIs to obtain and calculate the nutritional components of the food item.

[0363] Sentiment analysis utilizes automated text processing. Specifically, natural language processing libraries (such as NLTK or SpaCy) are used to analyze the sentiment from submitted text and evaluate whether the sentiment is positive or negative.

[0364] After this, the server will provide the user with optimal meal suggestions based on calculated nutritional content and emotional analysis. The selected suggestions will include meals that take key nutrients into consideration and foods that match the user's emotions. The user's device will be notified of meal or cooking suggestions, and these suggestions may include information on nearby partner stores based on location data.

[0365] For example, if a user posts an image to social media with the comment "A fun day of barbecuing," the server uses AI to analyze the image and identify the emotion as "joy." Based on this information, it becomes possible to suggest meals that promote a more positive experience, such as nutritious salads and fruit juices.

[0366] An example of a prompt to input into the generative AI model would be, "Please suggest a meal when a user posts an outdoor image with the comment 'It was a fun day!'" In this way, the goal is to provide a personalized experience that is tailored to the user's actions and emotions.

[0367] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0368] Step 1:

[0369] Users link their social media accounts to the system, allowing the server to access their posts. The input information consists of the user's account authentication credentials. This prepares the server to retrieve post data using the API. Specifically, authentication protocols such as OAuth are used to securely configure access rights.

[0370] Step 2:

[0371] The server receives image and text data posted by users via the SNS API. The input consists of newly posted images and text. As output, this data is stored within the system and used in the next analysis step. Specifically, the API call pulls the data and saves it to the server's storage.

[0372] Step 3:

[0373] The server utilizes image recognition AI to analyze the acquired image data. The input is the image data acquired in step 2. This analysis identifies food and dishes within the image and extracts their information. The output is the identified food name and ingredient information. Specifically, it runs an image classification model using TensorFlow or PyTorch to estimate the type and ingredients of the dish.

[0374] Step 4:

[0375] The server retrieves nutritional information by accessing nutrition databases and external APIs based on the food name and ingredient information. The input is the food information identified in step 3. The output is detailed nutritional component data for each food. Specifically, it executes queries to external databases to retrieve nutritional components related to the food.

[0376] Step 5:

[0377] The server uses natural language processing techniques to analyze user sentiment from posted text. The input is the text data obtained in step 2. The output is the analyzed sentiment information (e.g., positive, negative). Specifically, it uses NLTK or SpaCy to analyze the text and calculate the sentiment score.

[0378] Step 6:

[0379] The server selects dishes to suggest to the user based on nutritional and emotional information. Inputs are nutritional data and emotional information. Output is a list of recommended dishes. Specifically, it executes an algorithm that selects the optimal dish based on the user's nutritional status and emotional state.

[0380] Step 7:

[0381] The server sends the selected recipe suggestions to the user's device. The input is the list of dishes from step 6. The output is the suggestions received by the user. Specifically, a push notification is sent via a dedicated app or similar, displaying the suggested content on the user's screen.

[0382] Step 8:

[0383] The terminal displays the submitted suggestions to the user, providing information to help them make their next meal selection. Input is a suggestion notification from the server. Output is visual feedback to the user. Specifically, it displays dish details and selection reasons on a GUI, and provides additional information and links if necessary.

[0384] (Application Example 2)

[0385] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0386] In modern food services, providing appropriate meals based on individual dietary habits and health conditions remains challenging. In particular, there is a need for technology that effectively suggests meals that consider the user's nutritional balance and emotional state, and delivers them quickly. Furthermore, coordinating with external supply services to achieve a seamless ordering experience is also a challenge.

[0387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0388] In this invention, the server includes means for receiving user posts, identification means for identifying meals from images contained in the received posts, and means for calculating nutritional balance based on the identified meals. This enables seamless ordering of meal suggestions and delivery services tailored to the individual user's nutritional needs and preferences.

[0389] A "user" refers to an individual who receives meal suggestions using this system.

[0390] "Posting" refers to information, especially images and text, that a user uploads to an online information sharing service.

[0391] "Images" refer to the visual data included in a post and are a source of information for identifying the food.

[0392] "Meal" refers to food or dishes identified by image recognition technology.

[0393] "Nutritional balance" refers to the proportion of nutrients contained in a specific meal and is a standard used to evaluate a user's health status.

[0394] "Emotions" refer to the psychological state perceived from the user's posted content and images, and are factors that influence meal suggestions.

[0395] "Recommended meals" refer to meals recommended to users based on calculated nutritional balance and emotional state.

[0396] "External supply services" refer to third-party distributors or sellers who actually provide the proposed meals to users.

[0397] "Location information" refers to data that indicates the user's current location and is used to select suppliers that provide the suggested meals.

[0398] The system used to implement this application primarily consists of a server, a user terminal, and integration with external services. The server first receives SNS posts from the user's terminal. These posts contain images and text related to food.

[0399] The server analyzes food images acquired using image recognition technology to identify specific meals. This may involve using image recognition services such as the Google Cloud Vision API. Based on the identified meal, nutritional data is retrieved from an external nutrition information database or API. For example, the Nutritionix API could be used.

[0400] Furthermore, the server analyzes the text within the posts to determine the user's emotions. This analysis can utilize an emotion recognition engine such as IBM Watson Tone Analyzer. Based on the analysis results, it generates meal suggestions that take into account specific nutritional balances and emotional states.

[0401] Next, the server not only notifies the user of the suggested meal but also allows the user to order the suggested meal through an external supply service. It is possible to process orders by utilizing APIs of delivery services such as the Uber Eats API.

[0402] For example, if a user posts a photo of pancakes with the comment, "Sunday brunch was amazing!", the system might suggest a salad that is more nutritious compared to that snack. The suggested meal would then be presented to the user along with relevant nutritional information, and they could then order it.

[0403] An example of a prompt message might be: "Extract the name of the dish and the sentiment from the user's social media post, and suggest an appropriate meal based on that information. Check if the suggested dish is available for order through an existing delivery service."

[0404] Thus, this embodiment of the invention involves a server integrating and utilizing image recognition, emotion recognition, and nutritional analysis technologies to provide customized meal suggestions and ordering functions to individual users.

[0405] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0406] Step 1:

[0407] The server receives user posts via the SNS API. The input consists of user posts, which include image data and text information. This allows the server to import the received post data.

[0408] Step 2:

[0409] The server uses an image recognition engine to analyze the received image data. The input is the image data acquired in step 1, and the output is the identified meal or dish name. Here, the dish is identified from the image, and information is extracted for reference in the database.

[0410] Step 3:

[0411] The server sends a request to an external nutrition information API to retrieve nutritional data based on the identified meal. The input is the name of the dish, which is the output of step 2, and the output is the nutritional information associated with that dish. This provides the data necessary to calculate nutritional balance.

[0412] Step 4:

[0413] The server uses an emotion analysis engine to evaluate the user's emotions from the text information of the post. The input is the text information obtained in step 1, and the output is data indicating the user's emotional state. This clarifies the emotional indicators that should be considered when making meal suggestions.

[0414] Step 5:

[0415] The server combines nutritional balance data and emotional data to generate optimal meal suggestions for the user. The input is the output data from steps 3 and 4, and the output is the suggested meals. Calculations and selections are then performed to determine the appropriate meal.

[0416] Step 6:

[0417] The server notifies the user terminal of the suggested meal and confirms the possibility of ordering from an external supply service. The input is the suggested meal obtained in step 5, and the output is the notification to the user and ordering options. The user then receives the meal suggestion and can place an order immediately.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0419] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0421] [Third Embodiment]

[0422] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0423] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0424] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0426] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0428] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0429] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0430] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0433] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0434] This invention is a system that automatically analyzes nutritional information from food images posted by users on social media and suggests healthy meals. This system is implemented in the following specific form.

[0435] First, users link their social media accounts with this system. This allows the server to efficiently retrieve food photos that have been publicly shared from the user's account. The server periodically or in response to user actions uses social media APIs to collect posted image data.

[0436] Next, the server uses an AI module equipped with image recognition technology to analyze the acquired images. The analysis process identifies the type of food and related ingredients contained in the photo. For example, if the posted photo is of soup, the name of the dish, such as "minestrone" or "potage," will be identified, and vegetables and spices will be recognized as ingredients.

[0437] Subsequently, the server sends queries to a nutrition database or an external nutrition information API based on the identified ingredient information to retrieve the nutritional components of each ingredient. This allows the nutritional balance of the user's past meals to be calculated, and any deficient nutrients to be identified.

[0438] Based on the above analysis, the server selects dishes to suggest that supplement any missing nutrients. This may include menus and recipes from restaurants in the user's area. The selected dishes and ingredients will help improve the user's nutritional balance.

[0439] Finally, the server notifies the user of the selected suggestions on their device. Users can review the suggestions through the app or web interface on their device and use them to plan their next meal. The suggestions also include information on nearby partner restaurants, giving users more options to actually try the suggested dishes.

[0440] This invention will be implemented in a way that allows users to easily manage their daily dietary health and efficiently compensate for deficiencies in specific nutrients.

[0441] The following describes the processing flow.

[0442] Step 1:

[0443] Users link their social media accounts through the system's interface. This allows the server to gain access to the user's social media posts.

[0444] Step 2:

[0445] The server uses SNS APIs to periodically or at specified intervals retrieve image data related to cooking from user accounts. The data retrieved from SNS includes the image URL and associated text information.

[0446] Step 3:

[0447] The server runs an image recognition AI to analyze the acquired food images. Through this analysis, it identifies the type of dish and its ingredients, and generates the dish name and a list of ingredients.

[0448] Step 4:

[0449] The server uses the identified material information to access a nutrition database or an external nutrition information API to retrieve nutritional component data for each material.

[0450] Step 5:

[0451] Based on the acquired nutritional data, the server analyzes the nutritional balance calculated from the user's past meals, identifying necessary nutrients and nutrients in excess.

[0452] Step 6:

[0453] The server selects dishes to supplement any missing nutrients. This selection process takes into account menus from partner restaurants and general recipe information.

[0454] Step 7:

[0455] The server selects a proposal and prepares to notify the user's device and display it. The notification is sent via in-app messages or push notifications.

[0456] Step 8:

[0457] The device provides the user with an interface to review suggested dishes and ingredients, and the user receives and views information to help them choose their next meal.

[0458] (Example 1)

[0459] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0460] In recent years, healthy nutritional management has become increasingly important in individual diets, but it is difficult for individuals to accurately grasp the nutrients in the foods they consume daily and plan a balanced diet. Furthermore, the lack of nutritional information about food when eating out or during cooking makes it difficult for individuals to determine how to improve their balance, posing an obstacle to maintaining a healthy lifestyle.

[0461] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0462] In this invention, the server includes means for acquiring linked information via the user's information terminal, means for recognizing food using image processing technology based on the acquired information, and means for evaluating nutrients based on the recognized food information. This enables the automatic analysis of nutritional information of foods consumed daily by an individual and, based on the results, makes it possible to suggest meals that take nutritional balance into consideration.

[0463] A "user information terminal" refers to a device such as a mobile phone or computer used by a user, which is used to acquire information in conjunction with a system.

[0464] "Image processing technology" refers to the techniques used to analyze images represented as digital data and to understand and recognize their content.

[0465] "Means of recognizing food" refers to processes or systems for identifying food contained in acquired images and determining its type and attributes.

[0466] "Nutrient evaluation" is a method for analyzing the nutritional components of a specific food in detail and examining its nutritional value and impact on health.

[0467] "Food information" refers to detailed data related to a particular food product, such as its nutritional components, ingredients, and cooking methods.

[0468] "Nutritional balance" is an indicator that shows whether the various nutrients necessary for maintaining health are being consumed appropriately.

[0469] This invention is a system designed to support users in improving their eating habits. It utilizes images of food that users post on social media on a daily basis and automatically analyzes their nutritional information.

[0470] Users can begin analyzing food images by linking their SNS accounts with this system via their personal information terminal. Through this linkage, the system retrieves user-submitted data via the SNS API and sends the images to the server.

[0471] The server analyzes the collected images using image recognition modules such as Python's TensorFlow and PyTorch. This analysis process utilizes image processing techniques to identify the type of food and its ingredients. For example, if an image contains "pasta," the system will recognize ingredients such as "tomatoes" and "olive oil."

[0472] Next, the server sends queries to external nutrition databases and APIs to proceed with the process of obtaining nutritional information for the identified ingredients. Based on the nutritional data obtained from these databases, the system compares it with the user's past dietary records to evaluate nutritional balance. For example, if the system determines that the user is particularly deficient in "calcium," it will generate meal suggestions to compensate for that deficiency.

[0473] The suggested meals are generated by a generative AI model and associated with local restaurant information based on the user's location. This information is sent from the server to the user's device, and the user can view it through an application or web interface.

[0474] An example of a specific prompt is, "Analyze the food images posted by the user on social media, identify the nutritional information, and generate healthy meal suggestions." This prompt serves as an instruction to the generating AI model and is used to provide the user with suggestions based on the analysis results.

[0475] This system makes it easy for users to adopt a health-conscious diet and effectively supplement their nutritional needs.

[0476] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0477] Step 1:

[0478] Users link their social networking service (SNS) accounts to the system using an information terminal. The input required is the authentication information for the user's SNS account. Users grant access to the system and set the necessary API permissions through a dedicated application or web interface. This allows the server to access the user's public posts.

[0479] Step 2:

[0480] The server collects user-submitted image data using the SNS API. The input is the URL of the user's posted image. The server periodically sends requests to the API to check for the latest posts. This data is stored on the server as images of food.

[0481] Step 3:

[0482] The server inputs the collected image data into an image recognition module for analysis to identify food items. The input consists of stored image data of dishes. The server uses TensorFlow or PyTorch to identify the food items contained in the images. The output is a list of food names and ingredients.

[0483] Step 4:

[0484] The server queries nutrition databases and external APIs for information on the ingredients of identified foods and retrieves the relevant nutritional components. The input is a list of food names and ingredients. The server accesses the database and retrieves the nutrients for each ingredient as numerical values. The output is the nutritional information for each ingredient.

[0485] Step 5:

[0486] The server evaluates the user's nutritional balance based on acquired nutritional information. Inputs include nutritional information and the user's past meal history. The server identifies deficient nutrients and generates meal suggestions based on the results. Output is the evaluation of the user's nutritional balance.

[0487] Step 6:

[0488] The server uses a generative AI model to create meal suggestions for improving nutritional balance. The input is the user's nutritional balance assessment result. The server considers the user's geographical information and generates suggestions that combine locally appropriate meals and information on affiliated restaurants. The output is the suggested meal plan.

[0489] Step 7:

[0490] The server notifies the user's information terminal of the meal suggestions it has created. The input is the suggested meal plan. The user receives push notifications and alerts using an application on their terminal to confirm the suggestions. This allows the user to put their healthy meal plan into action.

[0491] (Application Example 1)

[0492] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0493] In modern life, people are required to efficiently manage their health amidst busy daily routines. However, it is difficult for the average consumer to analyze their daily diet in detail and accurately supplement the necessary nutrients. Furthermore, with the increasing use of eating out and delivery services, there is a lack of accurate information to help people choose healthy menus. Therefore, there is a need for a system that can automatically analyze nutritional information from food images posted on social media, and then provide healthy meal suggestions and immediately available cooking options based on the results.

[0494] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0495] In this invention, the server includes a mechanism for receiving posted information, a recognition mechanism for extracting cooking targets from image information, a mechanism for analyzing nutritional status based on the extracted cooking targets, a mechanism for presenting options for providing selected cooking targets in cooperation with nearby distribution services, and a mechanism for notifying the user of the recommended cooking targets. This enables users to accurately understand their nutritional status based on images obtained from social media and to receive healthy meals accordingly.

[0496] "Posted information" refers to information, including images and text, that users make public through social networking services, etc.

[0497] "Image information" refers to digital data that visually represents food or other objects.

[0498] "Items to be cooked" refers to individual ingredients or menu items that are served as a dish.

[0499] A "recognition mechanism" is a computer program or algorithm used to analyze image information and identify the food being cooked.

[0500] "Nutritional status" refers to the analysis results that show the types and amounts of nutrients contained in the food being cooked.

[0501] A "delivery service" refers to a delivery system or operator that provides food or ingredients to customers.

[0502] A "mechanism for presenting options" is a platform that provides users with available delivery menus and restaurant information based on analysis results.

[0503] A "notification mechanism" refers to a digital communication method or application used to notify users of information regarding proposed cooking targets and health improvements.

[0504] The system that realizes this invention consists of a server, a user's terminal, and an external API. First, an application installed on the user's terminal links with the SNS account. Through this link, the server can periodically retrieve the user's posted information via the SNS API.

[0505] The server uses a recognition mechanism to identify the food being cooked from image information in order to analyze the acquired post information. This recognition uses advanced image analysis software, TensorFlow and OpenCV, to identify specific dishes and ingredients. Subsequently, to analyze the nutritional status of the identified food, nutritional information is obtained from sources such as the USDA FoodData Central API, and a comprehensive nutritional analysis is performed.

[0506] Based on the analysis results, the server suggests healthy meals. This process involves collaborating with nearby delivery services to provide options based on the user's location. Delivery service APIs such as DoorDash and UberEats are used to present these options. Users can review the suggested options through their device's interface and place orders as needed.

[0507] As a concrete example, a Caesar salad might be identified from an image a user posted on social media, and based on an analysis of their past nutritional balance, the system might suggest adding grilled chicken to supplement their protein intake. The user can then easily order from a nearby restaurant based on this information.

[0508] An example of a prompt for a generative AI model is, "Please analyze the nutritional content of food images posted on social media and suggest healthy delivery menus."

[0509] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0510] Step 1:

[0511] The device links the SNS account with the application. When the user logs into the app and performs SNS authentication, the app and the SNS API are linked. The input requires the user's authentication information, and the output is permission to retrieve the user's posted information from the SNS.

[0512] Step 2:

[0513] The device uses the SNS API to send user-submitted images to the server. The server periodically collects submission information, particularly image data. The input in this step is image data, and the output is saving it to the image database on the server.

[0514] Step 3:

[0515] The server passes image data to a recognition mechanism to identify the food being cooked. TensorFlow and OpenCV are used for image recognition processing to analyze ingredients and dish names from the image. The input here is image data, and the output is a list of dish names and ingredients. Specifically, this involves AI-based image recognition.

[0516] Step 4:

[0517] The server retrieves nutritional information based on identified dishes and ingredients. It uses the USDA FoodData Central API to obtain nutritional data related to the dishes and analyze their nutritional status. Inputs include dish names and ingredient information, while output is a nutritional information table. Data processing includes sending queries to the API and analyzing the responses.

[0518] Step 5:

[0519] The server suggests healthy meals based on the analyzed nutritional information. It uses an AI model to generate options to ensure nutritional balance. The input is nutritional information, and the output is a list of suggested healthy menus. This includes generating prompts and providing them to the AI.

[0520] Step 6:

[0521] The server retrieves information about restaurants that offer the suggested menu using the user's location and a delivery service API. The input is the user's location and the suggested menu, and the output is a list of available restaurants. This involves specific API calls and response processing.

[0522] Step 7:

[0523] The terminal notifies the user of healthy meal suggestions and store information received from the server. The user can review and select from the suggested information. The input is the suggested information and store information, and the output is the notification presented to the user. Specific sending actions are included.

[0524] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0525] This invention is a system that analyzes food images posted on social media to evaluate a user's nutritional balance, suggests appropriate meals, and recognizes the user's emotions, reflecting them in the meal suggestions. This system is implemented as follows.

[0526] First, users link their social media accounts with this system, allowing the server to access their posts. The server uses the social media API to periodically retrieve the relevant food image data, or upon user request. The retrieved image data also includes the text information of the posts.

[0527] Next, the server uses image recognition AI to analyze the acquired food images. This analysis extracts the identified dish name and related ingredient information. Based on this, the server sends requests to a nutrition database or external nutrition information API to retrieve relevant nutritional components and calculate the user's nutritional balance.

[0528] The server then uses an emotion engine to evaluate the user's emotions based on the content and images of their posts. For example, if a user posts a photo with a positive comment, the emotion engine might recognize that emotion as "joy."

[0529] The server selects dishes to suggest based on calculated nutritional balance and perceived emotions. For users with positive emotions, it can suggest dishes that help maintain that mood, while for users with negative emotions, it can suggest dishes that help improve their mood.

[0530] Finally, the server notifies the user's device of the selected recommendations. The device displays the recommended dishes and the reasons for their selection, which the user can use to help with their next meal choice. The recommendations may also include information on nearby partner restaurants based on the user's location.

[0531] For example, if a user posts a photo of a summer barbecue with the comment "It was a fun day!", the server analyzes the nutritional information derived from the ingredients in the image and recognizes the emotion of "joy." Based on this, it suggests meals that promote further positive experiences, such as nutritious salads and fresh juices. In this way, by making meal suggestions that take the user's mood into consideration, more personalized health promotion becomes possible.

[0532] The following describes the processing flow.

[0533] Step 1:

[0534] Users link their social media accounts to this system. This grants the system access to their social media posts.

[0535] Step 2:

[0536] The server uses the SNS API to retrieve the user's latest posts. These posts include images of food uploaded by the user and the accompanying text.

[0537] Step 3:

[0538] The server uses an image processing module to analyze the acquired images and identify the type of dish and its main ingredients. AI technology is used to pinpoint the exact dish name and ingredients.

[0539] Step 4:

[0540] The server activates an emotion engine to analyze the user's emotional state from the content of the posted text and images. This analysis uses natural language processing techniques to recognize emotional expressions in the text.

[0541] Step 5:

[0542] The server uses the ingredient information of the identified dish to query a nutritional information database and retrieve nutritional data for each component. Based on this, it calculates the user's nutritional balance and analyzes any excesses or deficiencies.

[0543] Step 6:

[0544] The server selects a suitable meal for the user based on calculated nutritional balance and analyzed emotional state. The meal selection takes into account the user's mood and ensures that the necessary nutrients are provided.

[0545] Step 7:

[0546] The server selects a dish and notifies the user's device. The notification can include the reason for the suggestion, the nutritional characteristics of the dish, and its emotional impact.

[0547] Step 8:

[0548] The device provides the user with an interface that displays suggested dishes and their details. The user considers the suggestions and uses them as a reference when deciding what to eat next.

[0549] (Example 2)

[0550] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0551] Maintaining a nutritionally balanced diet is crucial for promoting health in modern life. However, providing nutritional information and meal suggestions tailored to each user's individual dietary habits and emotional state is challenging, and an efficient method is needed. Especially with the increasing number of users posting about their meals on social media, analyzing this information in real time and providing appropriate suggestions could contribute to users' ongoing health management.

[0552] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0553] In this invention, the server includes a device for receiving user submissions, an identification device for identifying food from image data contained in the received submissions, a device for calculating nutritional composition based on the identified food, a device for selecting food to suggest, a device for notifying the user of the suggested food, and a device for analyzing the user's emotions and reflecting them in the suggestions. This makes it possible to provide meal suggestions that simultaneously consider the user's nutritional status and emotional state.

[0554] A "device for receiving user posts" refers to hardware or software used to acquire digital data shared by users on social networking services (SNS) or online platforms.

[0555] A "food identification device from image data" is a device that uses image recognition technology to identify food or dishes shown in acquired image data.

[0556] A "device for calculating nutritional composition based on identified foods" is a device that quantifies the nutrients of identified foods and evaluates their balance based on the ingredients and component information of those foods.

[0557] A "device for selecting suggested foods" is a device that combines calculated nutritional information with emotional analysis data to select foods and dishes suitable for the user.

[0558] A "device that notifies users of proposed food items" is a device that has a system function for transmitting information about selected food items or dishes to the user's device.

[0559] A "device that analyzes user emotions and reflects them in suggestions" is a device that reads emotions from text and images and uses the identified emotions to provide personalized food suggestions.

[0560] This invention is implemented through a system that analyzes users' food posts and emotions, and makes suggestions based on the results. Users link their social media accounts to the system, allowing the server to receive data from these posts.

[0561] The server uses SNS APIs to periodically retrieve posted data, including text and images. For image data, it utilizes image recognition AI such as TensorFlow and PyTorch to identify food items and dishes. Based on this, it extracts information about the food's ingredients and then uses nutrition databases and nutrition information APIs to obtain and calculate the nutritional components of the food item.

[0562] Sentiment analysis utilizes automated text processing. Specifically, natural language processing libraries (such as NLTK or SpaCy) are used to analyze the sentiment from submitted text and evaluate whether the sentiment is positive or negative.

[0563] After this, the server will provide the user with optimal meal suggestions based on calculated nutritional content and emotional analysis. The selected suggestions will include meals that take key nutrients into consideration and foods that match the user's emotions. The user's device will be notified of meal or cooking suggestions, and these suggestions may include information on nearby partner stores based on location data.

[0564] For example, if a user posts an image to social media with the comment "A fun day of barbecuing," the server uses AI to analyze the image and identify the emotion as "joy." Based on this information, it becomes possible to suggest meals that promote a more positive experience, such as nutritious salads and fruit juices.

[0565] An example of a prompt to input into the generative AI model would be, "Please suggest a meal when a user posts an outdoor image with the comment 'It was a fun day!'" In this way, the goal is to provide a personalized experience that is tailored to the user's actions and emotions.

[0566] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0567] Step 1:

[0568] Users link their social media accounts to the system, allowing the server to access their posts. The input information consists of the user's account authentication credentials. This prepares the server to retrieve post data using the API. Specifically, authentication protocols such as OAuth are used to securely configure access rights.

[0569] Step 2:

[0570] The server receives image and text data posted by users via the SNS API. The input consists of newly posted images and text. As output, this data is stored within the system and used in the next analysis step. Specifically, the API call pulls the data and saves it to the server's storage.

[0571] Step 3:

[0572] The server utilizes image recognition AI to analyze the acquired image data. The input is the image data acquired in step 2. This analysis identifies food and dishes within the image and extracts their information. The output is the identified food name and ingredient information. Specifically, it runs an image classification model using TensorFlow or PyTorch to estimate the type and ingredients of the dish.

[0573] Step 4:

[0574] The server retrieves nutritional information by accessing nutrition databases and external APIs based on the food name and ingredient information. The input is the food information identified in step 3. The output is detailed nutritional component data for each food. Specifically, it executes queries to external databases to retrieve nutritional components related to the food.

[0575] Step 5:

[0576] The server uses natural language processing techniques to analyze user sentiment from posted text. The input is the text data obtained in step 2. The output is the analyzed sentiment information (e.g., positive, negative). Specifically, it uses NLTK or SpaCy to analyze the text and calculate the sentiment score.

[0577] Step 6:

[0578] The server selects dishes to suggest to the user based on nutritional and emotional information. Inputs are nutritional data and emotional information. Output is a list of recommended dishes. Specifically, it executes an algorithm that selects the optimal dish based on the user's nutritional status and emotional state.

[0579] Step 7:

[0580] The server sends the selected recipe suggestions to the user's device. The input is the list of dishes from step 6. The output is the suggestions received by the user. Specifically, a push notification is sent via a dedicated app or similar, displaying the suggested content on the user's screen.

[0581] Step 8:

[0582] The terminal displays the submitted suggestions to the user, providing information to help them make their next meal selection. Input is a suggestion notification from the server. Output is visual feedback to the user. Specifically, it displays dish details and selection reasons on a GUI, and provides additional information and links if necessary.

[0583] (Application Example 2)

[0584] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] In modern food services, providing appropriate meals based on individual dietary habits and health conditions remains challenging. In particular, there is a need for technology that effectively suggests meals that consider the user's nutritional balance and emotional state, and delivers them quickly. Furthermore, coordinating with external supply services to achieve a seamless ordering experience is also a challenge.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes means for receiving user posts, identification means for identifying meals from images contained in the received posts, and means for calculating nutritional balance based on the identified meals. This enables seamless ordering of meal suggestions and delivery services tailored to the individual user's nutritional needs and preferences.

[0588] A "user" refers to an individual who receives meal suggestions using this system.

[0589] "Posting" refers to information, especially images and text, that a user uploads to an online information sharing service.

[0590] "Images" refer to the visual data included in a post and are a source of information for identifying the food.

[0591] "Meal" refers to food or dishes identified by image recognition technology.

[0592] "Nutritional balance" refers to the proportion of nutrients contained in a specific meal and is a standard used to evaluate a user's health status.

[0593] "Emotions" refer to the psychological state perceived from the user's posted content and images, and are factors that influence meal suggestions.

[0594] "Recommended meals" refer to meals recommended to users based on calculated nutritional balance and emotional state.

[0595] "External supply services" refer to third-party distributors or sellers who actually provide the proposed meals to users.

[0596] "Location information" refers to data that indicates the user's current location and is used to select suppliers that provide the suggested meals.

[0597] The system used to implement this application primarily consists of a server, a user terminal, and integration with external services. The server first receives SNS posts from the user's terminal. These posts contain images and text related to food.

[0598] The server analyzes food images acquired using image recognition technology to identify specific meals. This may involve using image recognition services such as the Google Cloud Vision API. Based on the identified meal, nutritional data is retrieved from an external nutrition information database or API. For example, the Nutritionix API could be used.

[0599] Furthermore, the server analyzes the text within the posts to determine the user's emotions. This analysis can utilize an emotion recognition engine such as IBM Watson Tone Analyzer. Based on the analysis results, it generates meal suggestions that take into account specific nutritional balances and emotional states.

[0600] Next, the server not only notifies the user of the suggested meal but also allows the user to order the suggested meal through an external supply service. It is possible to process orders by utilizing APIs of delivery services such as the Uber Eats API.

[0601] For example, if a user posts a photo of pancakes with the comment, "Sunday brunch was amazing!", the system might suggest a salad that is more nutritious compared to that snack. The suggested meal would then be presented to the user along with relevant nutritional information, and they could then order it.

[0602] An example of a prompt message might be: "Extract the name of the dish and the sentiment from the user's social media post, and suggest an appropriate meal based on that information. Check if the suggested dish is available for order through an existing delivery service."

[0603] Thus, this embodiment of the invention involves a server integrating and utilizing image recognition, emotion recognition, and nutritional analysis technologies to provide customized meal suggestions and ordering functions to individual users.

[0604] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0605] Step 1:

[0606] The server receives user posts via the SNS API. The input consists of user posts, which include image data and text information. This allows the server to import the received post data.

[0607] Step 2:

[0608] The server uses an image recognition engine to analyze the received image data. The input is the image data acquired in step 1, and the output is the identified meal or dish name. Here, the dish is identified from the image, and information is extracted for reference in the database.

[0609] Step 3:

[0610] The server sends a request to an external nutrition information API to retrieve nutritional data based on the identified meal. The input is the name of the dish, which is the output of step 2, and the output is the nutritional information associated with that dish. This provides the data necessary to calculate nutritional balance.

[0611] Step 4:

[0612] The server uses an emotion analysis engine to evaluate the user's emotions from the text information of the post. The input is the text information obtained in step 1, and the output is data indicating the user's emotional state. This clarifies the emotional indicators that should be considered when making meal suggestions.

[0613] Step 5:

[0614] The server combines nutritional balance data and emotional data to generate optimal meal suggestions for the user. The input is the output data from steps 3 and 4, and the output is the suggested meals. Calculations and selections are then performed to determine the appropriate meal.

[0615] Step 6:

[0616] The server notifies the user terminal of the suggested meal and confirms the possibility of ordering from an external supply service. The input is the suggested meal obtained in step 5, and the output is the notification to the user and ordering options. The user then receives the meal suggestion and can place an order immediately.

[0617] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0618] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0620] [Fourth Embodiment]

[0621] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0622] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0623] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0625] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0627] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0628] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0629] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0630] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0633] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] This invention is a system that automatically analyzes nutritional information from food images posted by users on social media and suggests healthy meals. This system is implemented in the following specific form.

[0635] First, users link their social media accounts with this system. This allows the server to efficiently retrieve food photos that have been publicly shared from the user's account. The server periodically or in response to user actions uses social media APIs to collect posted image data.

[0636] Next, the server uses an AI module equipped with image recognition technology to analyze the acquired images. The analysis process identifies the type of food and related ingredients contained in the photo. For example, if the posted photo is of soup, the name of the dish, such as "minestrone" or "potage," will be identified, and vegetables and spices will be recognized as ingredients.

[0637] Subsequently, the server sends queries to a nutrition database or an external nutrition information API based on the identified ingredient information to retrieve the nutritional components of each ingredient. This allows the nutritional balance of the user's past meals to be calculated, and any deficient nutrients to be identified.

[0638] Based on the above analysis, the server selects dishes to suggest that supplement any missing nutrients. This may include menus and recipes from restaurants in the user's area. The selected dishes and ingredients will help improve the user's nutritional balance.

[0639] Finally, the server notifies the user of the selected suggestions on their device. Users can review the suggestions through the app or web interface on their device and use them to plan their next meal. The suggestions also include information on nearby partner restaurants, giving users more options to actually try the suggested dishes.

[0640] This invention will be implemented in a way that allows users to easily manage their daily dietary health and efficiently compensate for deficiencies in specific nutrients.

[0641] The following describes the processing flow.

[0642] Step 1:

[0643] Users link their social media accounts through the system's interface. This allows the server to gain access to the user's social media posts.

[0644] Step 2:

[0645] The server uses SNS APIs to periodically or at specified intervals retrieve image data related to cooking from user accounts. The data retrieved from SNS includes the image URL and associated text information.

[0646] Step 3:

[0647] The server runs an image recognition AI to analyze the acquired food images. Through this analysis, it identifies the type of dish and its ingredients, and generates the dish name and a list of ingredients.

[0648] Step 4:

[0649] The server uses the identified material information to access a nutrition database or an external nutrition information API to retrieve nutritional component data for each material.

[0650] Step 5:

[0651] Based on the acquired nutritional data, the server analyzes the nutritional balance calculated from the user's past meals, identifying necessary nutrients and nutrients in excess.

[0652] Step 6:

[0653] The server selects dishes to supplement any missing nutrients. This selection process takes into account menus from partner restaurants and general recipe information.

[0654] Step 7:

[0655] The server selects a proposal and prepares to notify the user's device and display it. The notification is sent via in-app messages or push notifications.

[0656] Step 8:

[0657] The device provides the user with an interface to review suggested dishes and ingredients, and the user receives and views information to help them choose their next meal.

[0658] (Example 1)

[0659] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] In recent years, healthy nutritional management has become increasingly important in individual diets, but it is difficult for individuals to accurately grasp the nutrients in the foods they consume daily and plan a balanced diet. Furthermore, the lack of nutritional information about food when eating out or during cooking makes it difficult for individuals to determine how to improve their balance, posing an obstacle to maintaining a healthy lifestyle.

[0661] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0662] In this invention, the server includes means for acquiring linked information via the user's information terminal, means for recognizing food using image processing technology based on the acquired information, and means for evaluating nutrients based on the recognized food information. This enables the automatic analysis of nutritional information of foods consumed daily by an individual and, based on the results, makes it possible to suggest meals that take nutritional balance into consideration.

[0663] A "user information terminal" refers to a device such as a mobile phone or computer used by a user, which is used to acquire information in conjunction with a system.

[0664] "Image processing technology" refers to the techniques used to analyze images represented as digital data and to understand and recognize their content.

[0665] "Means of recognizing food" refers to processes or systems for identifying food contained in acquired images and determining its type and attributes.

[0666] "Nutrient evaluation" is a method for analyzing the nutritional components of a specific food in detail and examining its nutritional value and impact on health.

[0667] "Food information" refers to detailed data related to a particular food product, such as its nutritional components, ingredients, and cooking methods.

[0668] "Nutritional balance" is an indicator that shows whether the various nutrients necessary for maintaining health are being consumed appropriately.

[0669] This invention is a system designed to support users in improving their eating habits. It utilizes images of food that users post on social media on a daily basis and automatically analyzes their nutritional information.

[0670] Users can begin analyzing food images by linking their SNS accounts with this system via their personal information terminal. Through this linkage, the system retrieves user-submitted data via the SNS API and sends the images to the server.

[0671] The server analyzes the collected images using image recognition modules such as Python's TensorFlow and PyTorch. This analysis process utilizes image processing techniques to identify the type of food and its ingredients. For example, if an image contains "pasta," the system will recognize ingredients such as "tomatoes" and "olive oil."

[0672] Next, the server sends queries to external nutrition databases and APIs to proceed with the process of obtaining nutritional information for the identified ingredients. Based on the nutritional data obtained from these databases, the system compares it with the user's past dietary records to evaluate nutritional balance. For example, if the system determines that the user is particularly deficient in "calcium," it will generate meal suggestions to compensate for that deficiency.

[0673] The suggested meals are generated by a generative AI model and associated with local restaurant information based on the user's location. This information is sent from the server to the user's device, and the user can view it through an application or web interface.

[0674] An example of a specific prompt is, "Analyze the food images posted by the user on social media, identify the nutritional information, and generate healthy meal suggestions." This prompt serves as an instruction to the generating AI model and is used to provide the user with suggestions based on the analysis results.

[0675] This system makes it easy for users to adopt a health-conscious diet and effectively supplement their nutritional needs.

[0676] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0677] Step 1:

[0678] Users link their social networking service (SNS) accounts to the system using an information terminal. The input required is the authentication information for the user's SNS account. Users grant access to the system and set the necessary API permissions through a dedicated application or web interface. This allows the server to access the user's public posts.

[0679] Step 2:

[0680] The server collects user-submitted image data using the SNS API. The input is the URL of the user's posted image. The server periodically sends requests to the API to check for the latest posts. This data is stored on the server as images of food.

[0681] Step 3:

[0682] The server inputs the collected image data into an image recognition module for analysis to identify food items. The input consists of stored image data of dishes. The server uses TensorFlow or PyTorch to identify the food items contained in the images. The output is a list of food names and ingredients.

[0683] Step 4:

[0684] The server queries nutrition databases and external APIs for information on the ingredients of identified foods and retrieves the relevant nutritional components. The input is a list of food names and ingredients. The server accesses the database and retrieves the nutrients for each ingredient as numerical values. The output is the nutritional information for each ingredient.

[0685] Step 5:

[0686] The server evaluates the user's nutritional balance based on acquired nutritional information. Inputs include nutritional information and the user's past meal history. The server identifies deficient nutrients and generates meal suggestions based on the results. Output is the evaluation of the user's nutritional balance.

[0687] Step 6:

[0688] The server uses a generative AI model to create meal suggestions for improving nutritional balance. The input is the user's nutritional balance assessment result. The server considers the user's geographical information and generates suggestions that combine locally appropriate meals and information on affiliated restaurants. The output is the suggested meal plan.

[0689] Step 7:

[0690] The server notifies the user's information terminal of the meal suggestions it has created. The input is the suggested meal plan. The user receives push notifications and alerts using an application on their terminal to confirm the suggestions. This allows the user to put their healthy meal plan into action.

[0691] (Application Example 1)

[0692] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0693] In modern life, people are required to efficiently manage their health amidst busy daily routines. However, it is difficult for the average consumer to analyze their daily diet in detail and accurately supplement the necessary nutrients. Furthermore, with the increasing use of eating out and delivery services, there is a lack of accurate information to help people choose healthy menus. Therefore, there is a need for a system that can automatically analyze nutritional information from food images posted on social media, and then provide healthy meal suggestions and immediately available cooking options based on the results.

[0694] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0695] In this invention, the server includes a mechanism for receiving posted information, a recognition mechanism for extracting cooking targets from image information, a mechanism for analyzing nutritional status based on the extracted cooking targets, a mechanism for presenting options for providing selected cooking targets in cooperation with nearby distribution services, and a mechanism for notifying the user of the recommended cooking targets. This enables users to accurately understand their nutritional status based on images obtained from social media and to receive healthy meals accordingly.

[0696] "Posted information" refers to information, including images and text, that users make public through social networking services, etc.

[0697] "Image information" refers to digital data that visually represents food or other objects.

[0698] "Items to be cooked" refers to individual ingredients or menu items that are served as a dish.

[0699] A "recognition mechanism" is a computer program or algorithm used to analyze image information and identify the food being cooked.

[0700] "Nutritional status" refers to the analysis results that show the types and amounts of nutrients contained in the food being cooked.

[0701] A "delivery service" refers to a delivery system or operator that provides food or ingredients to customers.

[0702] A "mechanism for presenting options" is a platform that provides users with available delivery menus and restaurant information based on analysis results.

[0703] A "notification mechanism" refers to a digital communication method or application used to notify users of information regarding proposed cooking targets and health improvements.

[0704] The system that realizes this invention consists of a server, a user's terminal, and an external API. First, an application installed on the user's terminal links with the SNS account. Through this link, the server can periodically retrieve the user's posted information via the SNS API.

[0705] The server uses a recognition mechanism to identify the food being cooked from image information in order to analyze the acquired post information. This recognition uses advanced image analysis software, TensorFlow and OpenCV, to identify specific dishes and ingredients. Subsequently, to analyze the nutritional status of the identified food, nutritional information is obtained from sources such as the USDA FoodData Central API, and a comprehensive nutritional analysis is performed.

[0706] Based on the analysis results, the server suggests healthy meals. This process involves collaborating with nearby delivery services to provide options based on the user's location. Delivery service APIs such as DoorDash and UberEats are used to present these options. Users can review the suggested options through their device's interface and place orders as needed.

[0707] As a concrete example, a Caesar salad might be identified from an image a user posted on social media, and based on an analysis of their past nutritional balance, the system might suggest adding grilled chicken to supplement their protein intake. The user can then easily order from a nearby restaurant based on this information.

[0708] An example of a prompt for a generative AI model is, "Please analyze the nutritional content of food images posted on social media and suggest healthy delivery menus."

[0709] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0710] Step 1:

[0711] The device links the SNS account with the application. When the user logs into the app and performs SNS authentication, the app and the SNS API are linked. The input requires the user's authentication information, and the output is permission to retrieve the user's posted information from the SNS.

[0712] Step 2:

[0713] The device uses the SNS API to send user-submitted images to the server. The server periodically collects submission information, particularly image data. The input in this step is image data, and the output is saving it to the image database on the server.

[0714] Step 3:

[0715] The server passes image data to a recognition mechanism to identify the food being cooked. TensorFlow and OpenCV are used for image recognition processing to analyze ingredients and dish names from the image. The input here is image data, and the output is a list of dish names and ingredients. Specifically, this involves AI-based image recognition.

[0716] Step 4:

[0717] The server retrieves nutritional information based on identified dishes and ingredients. It uses the USDA FoodData Central API to obtain nutritional data related to the dishes and analyze their nutritional status. Inputs include dish names and ingredient information, while output is a nutritional information table. Data processing includes sending queries to the API and analyzing the responses.

[0718] Step 5:

[0719] The server suggests healthy meals based on the analyzed nutritional information. It uses an AI model to generate options to ensure nutritional balance. The input is nutritional information, and the output is a list of suggested healthy menus. This includes generating prompts and providing them to the AI.

[0720] Step 6:

[0721] The server retrieves information about restaurants that offer the suggested menu using the user's location and a delivery service API. The input is the user's location and the suggested menu, and the output is a list of available restaurants. This involves specific API calls and response processing.

[0722] Step 7:

[0723] The terminal notifies the user of healthy meal suggestions and store information received from the server. The user can review and select from the suggested information. The input is the suggested information and store information, and the output is the notification presented to the user. Specific sending actions are included.

[0724] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0725] This invention is a system that analyzes food images posted on social media to evaluate a user's nutritional balance, suggests appropriate meals, and recognizes the user's emotions, reflecting them in the meal suggestions. This system is implemented as follows.

[0726] First, users link their social media accounts with this system, allowing the server to access their posts. The server uses the social media API to periodically retrieve the relevant food image data, or upon user request. The retrieved image data also includes the text information of the posts.

[0727] Next, the server uses image recognition AI to analyze the acquired food images. This analysis extracts the identified dish name and related ingredient information. Based on this, the server sends requests to a nutrition database or external nutrition information API to retrieve relevant nutritional components and calculate the user's nutritional balance.

[0728] The server then uses an emotion engine to evaluate the user's emotions based on the content and images of their posts. For example, if a user posts a photo with a positive comment, the emotion engine might recognize that emotion as "joy."

[0729] The server selects dishes to suggest based on calculated nutritional balance and perceived emotions. For users with positive emotions, it can suggest dishes that help maintain that mood, while for users with negative emotions, it can suggest dishes that help improve their mood.

[0730] Finally, the server notifies the user's device of the selected recommendations. The device displays the recommended dishes and the reasons for their selection, which the user can use to help with their next meal choice. The recommendations may also include information on nearby partner restaurants based on the user's location.

[0731] For example, if a user posts a photo of a summer barbecue with the comment "It was a fun day!", the server analyzes the nutritional information derived from the ingredients in the image and recognizes the emotion of "joy." Based on this, it suggests meals that promote further positive experiences, such as nutritious salads and fresh juices. In this way, by making meal suggestions that take the user's mood into consideration, more personalized health promotion becomes possible.

[0732] The following describes the processing flow.

[0733] Step 1:

[0734] Users link their social media accounts to this system. This grants the system access to their social media posts.

[0735] Step 2:

[0736] The server uses the SNS API to retrieve the user's latest posts. These posts include images of food uploaded by the user and the accompanying text.

[0737] Step 3:

[0738] The server uses an image processing module to analyze the acquired images and identify the type of dish and its main ingredients. AI technology is used to pinpoint the exact dish name and ingredients.

[0739] Step 4:

[0740] The server activates an emotion engine to analyze the user's emotional state from the content of the posted text and images. This analysis uses natural language processing techniques to recognize emotional expressions in the text.

[0741] Step 5:

[0742] The server uses the ingredient information of the identified dish to query a nutritional information database and retrieve nutritional data for each component. Based on this, it calculates the user's nutritional balance and analyzes any excesses or deficiencies.

[0743] Step 6:

[0744] The server selects a suitable meal for the user based on calculated nutritional balance and analyzed emotional state. The meal selection takes into account the user's mood and ensures that the necessary nutrients are provided.

[0745] Step 7:

[0746] The server selects a dish and notifies the user's device. The notification can include the reason for the suggestion, the nutritional characteristics of the dish, and its emotional impact.

[0747] Step 8:

[0748] The device provides the user with an interface that displays suggested dishes and their details. The user considers the suggestions and uses them as a reference when deciding what to eat next.

[0749] (Example 2)

[0750] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0751] Maintaining a nutritionally balanced diet is crucial for promoting health in modern life. However, providing nutritional information and meal suggestions tailored to each user's individual dietary habits and emotional state is challenging, and an efficient method is needed. Especially with the increasing number of users posting about their meals on social media, analyzing this information in real time and providing appropriate suggestions could contribute to users' ongoing health management.

[0752] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0753] In this invention, the server includes a device for receiving user submissions, an identification device for identifying food from image data contained in the received submissions, a device for calculating nutritional composition based on the identified food, a device for selecting food to suggest, a device for notifying the user of the suggested food, and a device for analyzing the user's emotions and reflecting them in the suggestions. This makes it possible to provide meal suggestions that simultaneously consider the user's nutritional status and emotional state.

[0754] A "device for receiving user posts" refers to hardware or software used to acquire digital data shared by users on social networking services (SNS) or online platforms.

[0755] A "food identification device from image data" is a device that uses image recognition technology to identify food or dishes shown in acquired image data.

[0756] A "device for calculating nutritional composition based on identified foods" is a device that quantifies the nutrients of identified foods and evaluates their balance based on the ingredients and component information of those foods.

[0757] A "device for selecting suggested foods" is a device that combines calculated nutritional information with emotional analysis data to select foods and dishes suitable for the user.

[0758] A "device that notifies users of proposed food items" is a device that has a system function for transmitting information about selected food items or dishes to the user's device.

[0759] A "device that analyzes user emotions and reflects them in suggestions" is a device that reads emotions from text and images and uses the identified emotions to provide personalized food suggestions.

[0760] This invention is implemented through a system that analyzes users' food posts and emotions, and makes suggestions based on the results. Users link their social media accounts to the system, allowing the server to receive data from these posts.

[0761] The server uses SNS APIs to periodically retrieve posted data, including text and images. For image data, it utilizes image recognition AI such as TensorFlow and PyTorch to identify food items and dishes. Based on this, it extracts information about the food's ingredients and then uses nutrition databases and nutrition information APIs to obtain and calculate the nutritional components of the food item.

[0762] Sentiment analysis utilizes automated text processing. Specifically, natural language processing libraries (such as NLTK or SpaCy) are used to analyze the sentiment from submitted text and evaluate whether the sentiment is positive or negative.

[0763] After this, the server will provide the user with optimal meal suggestions based on calculated nutritional content and emotional analysis. The selected suggestions will include meals that take key nutrients into consideration and foods that match the user's emotions. The user's device will be notified of meal or cooking suggestions, and these suggestions may include information on nearby partner stores based on location data.

[0764] For example, if a user posts an image to social media with the comment "A fun day of barbecuing," the server uses AI to analyze the image and identify the emotion as "joy." Based on this information, it becomes possible to suggest meals that promote a more positive experience, such as nutritious salads and fruit juices.

[0765] An example of a prompt to input into the generative AI model would be, "Please suggest a meal when a user posts an outdoor image with the comment 'It was a fun day!'" In this way, the goal is to provide a personalized experience that is tailored to the user's actions and emotions.

[0766] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0767] Step 1:

[0768] Users link their social media accounts to the system, allowing the server to access their posts. The input information consists of the user's account authentication credentials. This prepares the server to retrieve post data using the API. Specifically, authentication protocols such as OAuth are used to securely configure access rights.

[0769] Step 2:

[0770] The server receives image and text data posted by users via the SNS API. The input consists of newly posted images and text. As output, this data is stored within the system and used in the next analysis step. Specifically, the API call pulls the data and saves it to the server's storage.

[0771] Step 3:

[0772] The server utilizes image recognition AI to analyze the acquired image data. The input is the image data acquired in step 2. This analysis identifies food and dishes within the image and extracts their information. The output is the identified food name and ingredient information. Specifically, it runs an image classification model using TensorFlow or PyTorch to estimate the type and ingredients of the dish.

[0773] Step 4:

[0774] The server retrieves nutritional information by accessing nutrition databases and external APIs based on the food name and ingredient information. The input is the food information identified in step 3. The output is detailed nutritional component data for each food. Specifically, it executes queries to external databases to retrieve nutritional components related to the food.

[0775] Step 5:

[0776] The server uses natural language processing techniques to analyze user sentiment from posted text. The input is the text data obtained in step 2. The output is the analyzed sentiment information (e.g., positive, negative). Specifically, it uses NLTK or SpaCy to analyze the text and calculate the sentiment score.

[0777] Step 6:

[0778] The server selects dishes to suggest to the user based on nutritional and emotional information. Inputs are nutritional data and emotional information. Output is a list of recommended dishes. Specifically, it executes an algorithm that selects the optimal dish based on the user's nutritional status and emotional state.

[0779] Step 7:

[0780] The server sends the selected recipe suggestions to the user's device. The input is the list of dishes from step 6. The output is the suggestions received by the user. Specifically, a push notification is sent via a dedicated app or similar, displaying the suggested content on the user's screen.

[0781] Step 8:

[0782] The terminal displays the submitted suggestions to the user, providing information to help them make their next meal selection. Input is a suggestion notification from the server. Output is visual feedback to the user. Specifically, it displays dish details and selection reasons on a GUI, and provides additional information and links if necessary.

[0783] (Application Example 2)

[0784] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] In modern food services, providing appropriate meals based on individual dietary habits and health conditions remains challenging. In particular, there is a need for technology that effectively suggests meals that consider the user's nutritional balance and emotional state, and delivers them quickly. Furthermore, coordinating with external supply services to achieve a seamless ordering experience is also a challenge.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0787] In this invention, the server includes means for receiving user posts, identification means for identifying meals from images contained in the received posts, and means for calculating nutritional balance based on the identified meals. This enables seamless ordering of meal suggestions and delivery services tailored to the individual user's nutritional needs and preferences.

[0788] A "user" refers to an individual who receives meal suggestions using this system.

[0789] "Posting" refers to information, especially images and text, that a user uploads to an online information sharing service.

[0790] "Images" refer to the visual data included in a post and are a source of information for identifying the food.

[0791] "Meal" refers to food or dishes identified by image recognition technology.

[0792] "Nutritional balance" refers to the proportion of nutrients contained in a specific meal and is a standard used to evaluate a user's health status.

[0793] "Emotions" refer to the psychological state perceived from the user's posted content and images, and are factors that influence meal suggestions.

[0794] "Recommended meals" refer to meals recommended to users based on calculated nutritional balance and emotional state.

[0795] "External supply services" refer to third-party distributors or sellers who actually provide the proposed meals to users.

[0796] "Location information" refers to data that indicates the user's current location and is used to select suppliers that provide the suggested meals.

[0797] The system used to implement this application primarily consists of a server, a user terminal, and integration with external services. The server first receives SNS posts from the user's terminal. These posts contain images and text related to food.

[0798] The server analyzes food images acquired using image recognition technology to identify specific meals. This may involve using image recognition services such as the Google Cloud Vision API. Based on the identified meal, nutritional data is retrieved from an external nutrition information database or API. For example, the Nutritionix API could be used.

[0799] Furthermore, the server analyzes the text within the posts to determine the user's emotions. This analysis can utilize an emotion recognition engine such as IBM Watson Tone Analyzer. Based on the analysis results, it generates meal suggestions that take into account specific nutritional balances and emotional states.

[0800] Next, the server not only notifies the user of the suggested meal but also allows the user to order the suggested meal through an external supply service. It is possible to process orders by utilizing APIs of delivery services such as the Uber Eats API.

[0801] For example, if a user posts a photo of pancakes with the comment, "Sunday brunch was amazing!", the system might suggest a salad that is more nutritious compared to that snack. The suggested meal would then be presented to the user along with relevant nutritional information, and they could then order it.

[0802] An example of a prompt message might be: "Extract the name of the dish and the sentiment from the user's social media post, and suggest an appropriate meal based on that information. Check if the suggested dish is available for order through an existing delivery service."

[0803] Thus, this embodiment of the invention involves a server integrating and utilizing image recognition, emotion recognition, and nutritional analysis technologies to provide customized meal suggestions and ordering functions to individual users.

[0804] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0805] Step 1:

[0806] The server receives user posts via the SNS API. The input consists of user posts, which include image data and text information. This allows the server to import the received post data.

[0807] Step 2:

[0808] The server uses an image recognition engine to analyze the received image data. The input is the image data acquired in step 1, and the output is the identified meal or dish name. Here, the dish is identified from the image, and information is extracted for reference in the database.

[0809] Step 3:

[0810] The server sends a request to an external nutrition information API to retrieve nutritional data based on the identified meal. The input is the name of the dish, which is the output of step 2, and the output is the nutritional information associated with that dish. This provides the data necessary to calculate nutritional balance.

[0811] Step 4:

[0812] The server uses an emotion analysis engine to evaluate the user's emotions from the text information of the post. The input is the text information obtained in step 1, and the output is data indicating the user's emotional state. This clarifies the emotional indicators that should be considered when making meal suggestions.

[0813] Step 5:

[0814] The server combines nutritional balance data and emotional data to generate optimal meal suggestions for the user. The input is the output data from steps 3 and 4, and the output is the suggested meals. Calculations and selections are then performed to determine the appropriate meal.

[0815] Step 6:

[0816] The server notifies the user terminal of the suggested meal and confirms the possibility of ordering from an external supply service. The input is the suggested meal obtained in step 5, and the output is the notification to the user and ordering options. The user then receives the meal suggestion and can place an order immediately.

[0817] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0818] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0820] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0821] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0822] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0823] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0824] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0825] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0826] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0827] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0828] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0829] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0830] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0831] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0832] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0833] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0834] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0835] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0836] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0837] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0838] The following is further disclosed regarding the embodiments described above.

[0839] (Claim 1)

[0840] A means of receiving user posts,

[0841] An identification method for identifying a dish from an image included in a received post,

[0842] A means of calculating nutritional balance based on a specific dish,

[0843] A method for selecting dishes to be proposed based on a calculated nutritional balance,

[0844] A means of notifying the user of the above proposed dishes,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] A means of obtaining ingredient information related to a specific dish,

[0848] The means of obtaining nutritional components based on the material information further includes,

[0849] The system according to claim 1.

[0850] (Claim 3)

[0851] The means of obtaining information about restaurants that serve the suggested dishes based on the user's location information is further included.

[0852] The system according to claim 1.

[0853] "Example 1"

[0854] (Claim 1)

[0855] A means of obtaining linked information via the user's information terminal,

[0856] Based on the acquired information, a means for recognizing food using image processing technology,

[0857] A means of evaluating nutrients based on information about recognized foods,

[0858] A means of selecting foods recommended to supplement fire or various nutrients,

[0859] A means of providing users with information on selected food products,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] Means for obtaining attribute information related to recognized food,

[0863] The means of obtaining nutritional composition based on said attribute information is further included.

[0864] The system according to claim 1.

[0865] (Claim 3)

[0866] The system further includes means for obtaining food information based on the user's geographical location.

[0867] The system according to claim 1.

[0868] "Application Example 1"

[0869] (Claim 1)

[0870] A mechanism for receiving posted information,

[0871] A recognition mechanism for extracting the object to be cooked from image information,

[0872] A mechanism for analyzing nutritional status based on the extracted cooked food,

[0873] A mechanism for selecting recommended cooking targets based on analyzed nutritional status,

[0874] A mechanism that presents options for selecting and providing cooked food in cooperation with nearby distribution services,

[0875] A mechanism to notify users of the recommended cooking targets mentioned above,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] A mechanism for extracting relevant material information,

[0879] The mechanism further includes a mechanism for obtaining nutritional composition based on the material information.

[0880] The system according to claim 1.

[0881] (Claim 3)

[0882] The system further includes a mechanism for collecting location information to supply the proposed cooked food based on the user's location information.

[0883] The system according to claim 1.

[0884] "Example 2 of combining an emotion engine"

[0885] (Claim 1)

[0886] A device that receives user submissions,

[0887] An identification device that identifies food from image data contained in a received post,

[0888] A device that calculates nutritional composition based on identified foods,

[0889] A device that selects food items to be proposed based on the calculated nutritional composition,

[0890] A device that notifies the user of the food products to be suggested,

[0891] A device that analyzes user emotions and reflects them in suggestions,

[0892] A system that includes this.

[0893] (Claim 2)

[0894] A device for acquiring material information related to identified food products,

[0895] The apparatus further includes a device for obtaining nutritional components based on the said material information.

[0896] The system according to claim 1.

[0897] (Claim 3)

[0898] The device further includes a device that acquires information on sales offices that provide the proposed food products based on the user's location information.

[0899] The system according to claim 1.

[0900] "Application example 2 when combining with an emotional engine"

[0901] (Claim 1)

[0902] A means of receiving user posts,

[0903] An identification method for identifying a meal from an image included in a received post,

[0904] A means of calculating nutritional balance based on a specified diet,

[0905] A method for selecting suggested meals that take into account a calculated nutritional balance and the user's feelings,

[0906] A means of notifying the user of the suggested meals and ordering the suggested meals from an external supply service,

[0907] A system that includes this.

[0908] (Claim 2)

[0909] Means for obtaining component information related to identified meals,

[0910] The means of obtaining nutritional components based on the said component information further includes,

[0911] The system according to claim 1.

[0912] (Claim 3)

[0913] The means of obtaining information about suppliers that provide the suggested meals based on the user's location information is further included.

[0914] The system according to claim 1. [Explanation of Symbols]

[0915] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving user posts, An identification method for identifying a dish from an image included in a received post, A means of calculating nutritional balance based on a specific dish, A method for selecting dishes to be proposed based on a calculated nutritional balance, A means of notifying the user of the above proposed dishes, A system that includes this.

2. A means of obtaining ingredient information related to a specific dish, The means of obtaining nutritional components based on the material information further includes, The system according to claim 1.

3. The means of obtaining information about restaurants that serve the suggested dishes based on the user's location information is further included. The system according to claim 1.

Images