system

Abstract

Provide a system. 【Solution means】 Image analysis means for receiving and analyzing an image of a meal taken; Food component calculation means for identifying food from the analyzed image, obtaining the salt content thereof from a database, and summing them up; Notification generation means for transmitting the calculated salt content to a notification device; Warning issuance means for issuing a warning for a salt content exceeding a standard; Data accumulation means for accumulating analysis results and calculation results and storing long-term meal data; Visualization means for displaying the long-term meal salt content of a user in a chart format; Remote access means for enabling information sharing by multiple users; A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Hypertension in the elderly is particularly difficult to review lifestyle habits, and excessive salt intake in daily diet is often the main cause. In particular, the elderly living alone tend to unconsciously over-consume salt according to their taste preferences and are considered difficult to self-manage. In addition, it is difficult for relatives living far away to grasp information on the diet of the elderly in real time, which may result in losing the opportunity to provide appropriate advice. To address such problems, it is necessary to automatically visualize the amount of salt consumed daily and share the information with relatives to lead to an improvement in the diet.

Means for Solving the Problems

[0005] This invention includes an image analysis means for receiving and analyzing images of meals, a salt content calculation means for identifying food from the analyzed images and retrieving and summing its salt content from a database, a notification generation means for transmitting the calculated salt content to a notification device, and a warning issuing means for issuing a warning if the salt content exceeds a certain standard. Furthermore, it includes a data storage means for accumulating analysis results and calculation results and storing long-term meal data, thereby supporting the user's dietary management. In addition, it includes a relative notification means for notifying remote relatives of the captured images for review, allowing relatives to quickly identify excessive salt intake and provide appropriate advice. Moreover, it includes a data analysis means that enables analysis of the user's salt intake over time, enabling long-term health management.

[0006] "Image analysis means" refers to a device or method that receives and analyzes images of a photographed meal.

[0007] A "salt content calculation means" is a device or method that acquires salt content data for identified foods and calculates the total salt content by summing it up.

[0008] A "notification generation means" is a device or method that generates a notification message based on the calculated amount of salt and sends it to a related device.

[0009] A "warning issuance method" refers to a device or method that issues a warning when the salt content exceeds the standard.

[0010] "Data storage means" refers to devices or methods for storing analysis results and calculation results over a long period of time.

[0011] A "relative notification means" refers to a device or method that notifies remote relatives of captured images and related information, allowing them to review and confirm them.

[0012] "Data analysis means" refers to devices or methods that enable the analysis of the amount of salt a user has consumed over time. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The system according to the present invention helps elderly people living alone to easily manage their salt intake, and a specific embodiment thereof will be described below.

[0035] The system starts when the user takes a picture of their meal with a smartphone or dedicated device. The captured image is saved on the device and sent to a server via the network. Upon receiving the image, the server uses image analysis to identify the food items. A deep learning model is used here to label the different food items in the image.

[0036] For each identified food item, the server retrieves information on the corresponding salt content from its internal database and uses a salt content calculation device to sum these values ​​and calculate the total salt content. This calculation result is then used by a notification generation device to generate a notification message to be sent to the user and their relatives who are remotely connected. This message includes the salt content for each food item and the total salt content.

[0037] If the salt intake exceeds a pre-set standard, the server sends a signal to the terminal via a warning system, instructing the terminal to sound a warning. Furthermore, the analysis and calculation results are stored in the server's database as the user's meal history by a data storage system.

[0038] Furthermore, by analyzing this accumulated data using data analysis tools, it becomes possible to help users manage their health in the long term. This allows users to easily understand the temporal trends in their salt intake, and relatives can remotely provide dietary advice.

[0039] As described above, the present invention allows users to easily manage the amount of salt they consume from their daily meals, and furthermore, enables relatives to provide appropriate advice based on that information. This makes it possible to provide a system that supports the maintenance of the health of the elderly.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] Users take photos of their meals using their smartphones or dedicated devices. The captured images are automatically saved to the device.

[0043] Step 2:

[0044] The device uploads the saved photos to a designated cloud server. The photo data is transferred using an internet connection during the upload process.

[0045] Step 3:

[0046] The server activates an image analysis system to analyze the received photo data. This process uses deep learning technology to identify each food item in the image.

[0047] Step 4:

[0048] The server retrieves salt content information for each identified food item from its internal database. It aggregates the salt content data for each food item and calculates the total salt content using a salt content calculation device.

[0049] Step 5:

[0050] The server uses the calculated total salt content and the salt content information of individual foods to generate a notification message. This notification message is created for the user and their relatives.

[0051] Step 6:

[0052] The server sends the generated notification message to the user's and their relatives' devices via a notification generation mechanism. Registered contact information is used for sending the message.

[0053] Step 7:

[0054] If the salt content exceeds a set standard, the server sends an instruction to the terminal using a warning system, and the terminal notifies the user by sounding a warning.

[0055] Step 8:

[0056] The server will save the analysis and calculation results in a data storage system. This data will be stored as the user's long-term dietary history.

[0057] Step 9:

[0058] The server uses the accumulated data to activate data analysis tools and perform an analysis of salt intake over time. The results of this analysis are provided periodically to the user and their relatives.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Diet has a significant impact on health, and excessive salt intake, in particular, can increase health risks for the elderly. However, for the elderly, who increasingly eat alone, managing their daily salt intake appropriately is a difficult problem. Furthermore, even if relatives living far away want to support their daily diet, physical distance is often an obstacle. To address this, there is a need for a system that allows the elderly to easily and effectively manage their diet, and that enables relatives living far away to provide support.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes an analysis device means that receives a meal image captured from a location information terminal and detects food components by digital image processing; a nutritional analysis device means that extracts the amount of each component of the meal item from a data storage device based on the detected meal item and calculates its sum; and an information notification device means that transmits the calculated sum of component amounts to the user via a communication device. This allows the user to easily manage their daily salt intake, and enables relatives to remotely provide effective support for their diet.

[0064] A "location information terminal" is an electronic device used to capture images of food or other content and transmit them to a server.

[0065] "Digital image processing" refers to a technical method that analyzes received image data to detect food components.

[0066] "Food components" refer to the individual nutritional elements and their attributes contained in a meal.

[0067] An "analysis device" is a general term for hardware or software that analyzes the components and characteristics of food based on received data.

[0068] A "nutrition analyzer" is a device that calculates the amount of each component as a numerical value based on the detected food components and then calculates the sum of those values.

[0069] A "data storage device" refers to a database that stores information related to food ingredients and allows for searching and retrieving this information as needed.

[0070] An "information notification device" is a means of presenting information on the calculated sum of component amounts to the user, and functions as a communication device.

[0071] A "signal issuing device" is a device that generates alarms based on set criteria values ​​to alert the user.

[0072] An "information storage device" is a device that stores the results of analysis and calculations over a long period of time and manages historical data on food intake.

[0073] A "remote communication device" is a device or system that can transmit captured image information to an individual or group located at a distance.

[0074] A "data processing device" refers to a device that analyzes accumulated information and generates detailed reports based on changes over time.

[0075] This invention provides a system that allows elderly people living alone to easily manage their daily salt intake. The system starts when the user uses a location-based terminal to take a picture of their meal, saves the image to the terminal, and transmits it to a server via a network. The server analyzes the received image using digital image processing to detect food components. Various deep learning models specialized in image analysis are used for this purpose.

[0076] The server extracts component amounts from data storage based on the components obtained by the analysis device. These component amounts are then summed up using a nutritional analyzer to calculate the total salt intake. The calculated results are reported to the user via an information notification device. The notification includes the component amounts for each food item and the total component amounts, allowing the user to accurately control their intake.

[0077] If a user's salt intake exceeds a set limit, the server issues an alarm via a signaling device. Upon receiving the alarm, the terminal emits an audible or visual warning to alert the user. Analysis and calculation results are stored in an information storage device, and based on this historical data, the server uses a data processing device to analyze long-term nutritional intake trends and provide advice for maintaining health.

[0078] Furthermore, the captured image information can be shared with relatives in distant locations via a remote communication device, allowing them to provide direct support remotely. This sharing of information enables relatives to understand the user's daily eating habits and provide appropriate feedback.

[0079] A concrete example is a scenario where a user eats ramen and salad for lunch and takes a picture of it with their device. The server analyzes this image, identifies the salt content of each food item, and notifies the user of the results. The prompt used for the generating AI model is, "Take pictures of your ramen and salad for lunch and calculate the salt content of each." Through this prompt, the server can perform appropriate image analysis and provide information.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The user takes a picture of their meal using a smartphone or a dedicated camera device. The captured image is saved to the device. This image data becomes the input for the next process. Specifically, the user launches the camera app on their device, frames the meal, and presses the shutter button.

[0083] Step 2:

[0084] The terminal sends the saved image to the server over the network. Here, the input is the image file stored on the terminal, and the output is the image data received by the server. The terminal uses its communication function to upload the image file to the server according to the appropriate communication protocol.

[0085] Step 3:

[0086] The server analyzes the received image data using digital image processing technology. In this step, the image data transferred to the server is used as input, and label data identifying the food is generated as output. The server uses a generative AI model to extract the features of the food and label it.

[0087] Step 4:

[0088] The server searches the data storage device based on the analyzed label data to obtain the salt content of each food item. The input is food label data, and the output is the corresponding salt content information. The server queries the database using the name of the identified food item as a key to retrieve the corresponding salt content.

[0089] Step 5:

[0090] The server calculates the total salt content by summing up the salt content obtained using a nutritional analyzer. In this step, the salt content information of each food item is used as input, and the output is the calculated total salt content. Specifically, the process involves summing up the total salt content using numerical calculations.

[0091] Step 6:

[0092] The server notifies the user via an information notification device based on the total salt content result. The input is the calculated total salt content, and the output is a notification message displayed on the user's terminal. The server generates the notification content and sends the message to the user.

[0093] Step 7:

[0094] The server determines whether the calculated total salt content exceeds the set standard, and if so, issues an alarm via a signaling device. The inputs are the total salt content and the set standard, and the output is a warning signal. The server compares the value to the standard and, if necessary, generates a warning signal and sends it to the terminal.

[0095] Step 8:

[0096] The server stores analysis and calculation results in an information storage device and records them as long-term food intake history data. The input is a series of analysis and calculation results, and the output is historical data stored in a database. The server stores the information through a data management system.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Managing salt intake in meals smoothly for elderly people living alone is a challenge. Existing methods make it difficult to effectively utilize information even when photos of meals are taken, requiring continuous support from family members or caregivers. Furthermore, the inability to visually manage salt intake poses a risk of problems. Therefore, there is a need for a system that allows users to easily understand their salt intake and manage their health while sharing information with relatives.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes image analysis means for receiving and analyzing images of photographed meals, visualization means for displaying the user's long-term dietary salt intake in a chart format, and remote access means for enabling information sharing among multiple users. This makes it possible for users to easily manage their salt intake and manage their health while sharing information with relatives.

[0102] "Image analysis means" refers to a process used to receive images of a photographed meal and identify identifiable food items.

[0103] The "salt content calculation method" is a process that retrieves the salt content of food items identified based on the analyzed images from a database and then sums up the total salt content.

[0104] The "notification generation method" is a procedure for notifying the user's device of the calculated amount of salt.

[0105] A "warning signaling mechanism" is a system that issues a warning signal when the salt content exceeds the standard.

[0106] A "data storage method" is a process for storing analysis results and calculation results over a long period of time so that they can be analyzed later.

[0107] "Visualization means" refers to a function that displays the user's long-term salt intake data in a chart format.

[0108] "Remote access means" refers to technologies that enable multiple users to share and access information via the internet.

[0109] "Family notification method" refers to a method of notifying the user's relatives of the photographed meal images so that they can view them remotely.

[0110] "Data analysis tools" refer to tools used to analyze users' salt intake over time.

[0111] This invention provides a system that allows elderly people to easily manage their salt intake through meals. The system begins when a user takes a picture of their meal using a smartphone or other device. The device sends the captured image to a server, which then performs image analysis on the received image. The image analysis utilizes a deep learning model, enabling automatic identification of food items. For each identified food item, the server retrieves the salt content from a database and calculates the total salt intake.

[0112] The calculation results are notified to the user's device and to relatives located remotely. The notification includes information on the amount of salt in each food item and the total amount of salt. If the salt intake exceeds a pre-set standard, a signal is sent to the device to emit a warning sound. Furthermore, the server stores the analysis and calculation results as data for use in long-term health management. This stored data is visualized in various charts and graphs, making it easy to check salt intake over time. This also allows relatives to provide dietary advice remotely.

[0113] As a concrete example, by using a prompt such as, "Take a picture of your lunch, is today's salt intake okay?", users can easily enjoy the benefits of the invention. This system uses software such as Python and TENSORFLOW® to process and calculate data. Smartphones and tablets are commonly used as hardware. In this way, the present invention provides comprehensive support for easily managing the amount of salt consumed from daily meals.

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] A user takes a picture of their meal using their smartphone camera. The captured image is saved on the device and then sent to a server via the internet. In this process, the image is the input, and the data transmission to the server is the output.

[0117] Step 2:

[0118] The server analyzes the received images using a deep learning model. Specifically, each food item in the image is identified. A list of labeled food items is output from the image data. An object recognition algorithm is used in this process.

[0119] Step 3:

[0120] The server retrieves the salt content of each food item from the database based on the label. It takes a list of foods as input and outputs the corresponding salt content information. Data retrieval is performed here using SQL queries, etc.

[0121] Step 4:

[0122] The salt content calculation method calculates the total salt content by summing up the acquired salt amounts. An array containing the individual salt amounts is taken as input, and the total salt content is output. The calculation is performed using standard addition.

[0123] Step 5:

[0124] The calculated total salt intake is sent as a notification to the user's device and their relatives' devices. The notification message is output using the total salt intake data as input. A messaging API is used for this process.

[0125] Step 6:

[0126] The server stores the calculation results in a database, accumulating them as a long-term dietary history. Here, the analysis results and calculation results serve as input, and the accumulated data records are output. A database management system is used for data storage.

[0127] Step 7:

[0128] The system visualizes accumulated data in graphs and charts, allowing users to review their past salt intake history. Historical data retrieved from the database is used as input, and visualized information is output. Data analysis software is used for visualization.

[0129] Step 8:

[0130] If the user-set threshold is exceeded, a signal is sent to the device to emit a warning sound. The total salt content is used as input to output the warning sound. This process utilizes the device's notification system.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention aims to more effectively manage the health of the elderly by combining a salt management system for meals with a user emotion recognition function. Specific embodiments are described below.

[0133] The system initiates the process by having the user take a picture of their meal with their smartphone or a dedicated device. The captured image is saved on the device and uploaded to the server. The server uses image analysis to identify the food in the received image and retrieves salt content information from a database for each recognized food item.

[0134] Subsequently, the server uses a salt content calculation device to sum up the identified salt amounts and calculate the total salt content. The calculation result is sent as a message to the user and their relatives by a notification generation device. This message includes the salt content of individual foods and an indicator tailored to the newly adopted emotion.

[0135] The emotion engine, a key feature of this invention, has the ability to recognize the user's emotions in real time. The user's emotions are captured through camera or voice input and analyzed by an emotion recognition algorithm. Based on the results of this emotion analysis, notifications and warnings regarding salt intake are incorporated. For example, if the user is feeling stressed, the notification is also sent to relatives, including advice that takes stress reduction into consideration.

[0136] Furthermore, emotional data is stored in a data storage system along with the user's dietary history. This data is used by data analysis tools to analyze the user's salt intake over time, supporting long-term health management. In addition, fluctuations based on the user's emotional data are recorded over a long period and provided to family members. This enables comprehensive health management, including the user's psychological state.

[0137] For example, if a user feels anxious during a meal, the system recognizes this through an emotion engine. The server analyzes this and sends advice to the family, along with a notification about the amount of salt consumed, suggesting ways to relax and alleviate the anxiety. Based on this, the family can provide appropriate support to the user.

[0138] Thus, the present invention provides a system that supports a healthier lifestyle for the elderly by simultaneously managing salt intake and the user's emotions.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The user takes a photo of their meal using their smartphone or a dedicated device. This triggers the system to start operating.

[0142] Step 2:

[0143] The device temporarily stores the captured images and uploads them to the server via the network. The device is configured to automatically upload images after they are captured.

[0144] Step 3:

[0145] The server analyzes the received images based on image analysis tools to identify the food items in the photographs. AI-based image recognition technology is used for food identification.

[0146] Step 4:

[0147] The server retrieves the salt content of each food item from a database based on the data of the identified food item. This makes it possible to collect accurate salt information.

[0148] Step 5:

[0149] The server uses a salt content calculation device to add up the salt content of each food item and calculate the total salt content. This value will be used in subsequent notifications.

[0150] Step 6:

[0151] To recognize the user's emotions in real time, the device uses a camera or microphone to acquire emotional data from the user's facial expressions and voice.

[0152] Step 7:

[0153] Emotional data acquired from the terminal is sent to a server, where an emotion engine processes it and determines the user's emotions.

[0154] Step 8:

[0155] The server combines salt levels and emotional data to generate a notification message. This message includes advice based on the salt level and the corresponding emotional state.

[0156] Step 9:

[0157] The server sends the generated notifications to the user's and registered relatives' devices. It provides not only salt intake calculation results but also emotionally-driven advice.

[0158] Step 10:

[0159] If the server detects a salt level exceeding the standard or any resulting emotional changes, it will use a warning system on the terminal to emit a warning sound or message.

[0160] Step 11:

[0161] The server stores analysis results and emotional data in a data storage system. This data is used for long-term health management and tracking changes in the user's emotional state.

[0162] Step 12:

[0163] The stored data is analyzed using data analysis tools. This makes it easier to understand the user's health status, salt intake, and emotional state trends.

[0164] This system allows for the simultaneous management of the dietary habits and emotional well-being of elderly individuals, enabling more comprehensive health management.

[0165] (Example 2)

[0166] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0167] For elderly individuals, managing salt intake is crucial for maintaining health. However, accurately determining the salt content of meals is generally difficult. Furthermore, the impact of emotional state during meals on health should also be considered. Conventional systems have not comprehensively considered both salt management and the user's emotional state. Therefore, there is a need for technology that can simultaneously achieve appropriate management of dietary content and care for emotional state.

[0168] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0169] In this invention, the server includes an image processing means for receiving and analyzing images of photographed meals, a quantity calculation means for identifying food from the analyzed images and obtaining and summing the salt content from an information storage device, and an emotion analysis means for recognizing the user's emotions and reflecting them in the analysis results and notifications. This enables comprehensive health management that accurately manages the amount of salt in the user's meals while also taking into account their emotional state.

[0170] "Image processing means" refers to technology that provides functions for receiving images of photographed food and analyzing those images.

[0171] The "quantity calculation means" is a technology for obtaining and summing the salt content of food identified from the analyzed images by retrieving it from an information storage device.

[0172] "Notification generation means" refers to a technology equipped with a function for transmitting calculated salt content information to a notification device.

[0173] A "warning issuance method" is a technology that provides a method for issuing a warning when the salt content exceeds the standard.

[0174] "Information storage means" refers to technology for accumulating analysis results and calculation results to store dietary information over a long period of time.

[0175] "Emotion analysis means" refers to technology that recognizes the user's emotions and reflects them in the analysis results and notifications.

[0176] "Notification adjustment means" refers to technology that has a function to adjust notification content based on the user's emotional state.

[0177] "Family notification means" refers to a technology that notifies a remote relative of a captured image, allowing that relative to view it.

[0178] "Information analysis means" refers to technology for performing analysis based on the amount of salt consumed by the user and the time course of their emotional state.

[0179] This invention is a system that supports health management by comprehensively analyzing a user's dietary management and emotional state. Specific embodiments are described below.

[0180] Users take photos of their meals using their smartphones or dedicated terminals. These terminals store the images and upload them to a server via the network. The server analyzes the received images using image processing software (e.g., OpenCV or TensorFlow) to identify the food items. Machine learning algorithms are used in this process, based on the food's shape, color, patterns, and other characteristics.

[0181] The server calculates the salt content using information about the identified food items. This information is obtained from a nutritional database. The server then sends the total salt content to the user and distant relatives via a notification generation system. This notification includes details of the meal and the total salt content.

[0182] Furthermore, the user's emotions are captured using the device's camera and microphone. These are processed in real time using emotion analysis algorithms (for example, Amazon Rekognition or Google® Cloud Speech-to-Text). The server adjusts the notification content based on the recognized emotions and sends it to relatives, including appropriate advice.

[0183] This system stores users' dietary history and emotional data using information storage mechanisms for long-term use. The data is analyzed using information analysis mechanisms to track changes in the user's salt intake and emotional state over time. This enables comprehensive health management.

[0184] For example, if a user feels down while eating a fish dish at dinner, the system recognizes this situation and sends a notification to a relative that includes advice on how to improve their mood, along with information on the amount of salt consumed.

[0185] An example of a prompt to input into the generative AI model is as follows: "I would like to detail the specific processes of a health management system for the elderly. This includes everything from taking photos of meals to managing salt intake, and also mentions the role of sentiment analysis."

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The user takes a picture of their meal using a smartphone or dedicated device. The captured image is saved to the device. The input is image data generated by the user, and the output is an image file stored on the device. Specifically, the camera application is used to capture the image and save it to the device's storage.

[0189] Step 2:

[0190] The terminal uploads the saved image to the server via the network. The input is the image file saved in step 1, and the output is the image data transferred to the server. Specifically, the image file is sent to a specified endpoint on the server via the internet connection.

[0191] Step 3:

[0192] The server analyzes the received images through image processing tools. It uses software such as OpenCV and TensorFlow to identify food items within the images. The input is image data uploaded to the server, and the output is identification information for the food items in the images. Specifically, it runs a machine learning model to analyze shape, color, and pattern to identify the food items.

[0193] Step 4:

[0194] The server retrieves the salt content of identified foods from a database and sums them up. The input is a list of identified foods, and the output is the salt content of each food and the sum of those salt content values. Specifically, it sends a query to the database for each food and adds up the retrieved salt content data.

[0195] Step 5:

[0196] The server generates a message containing the total salt content and food details, and sends it to the user and their relatives. The input is the calculated total salt content and food details, and the output is the notification message. Specifically, it uses a notification generation engine to create and send emails and app notifications.

[0197] Step 6:

[0198] The device recognizes the user's emotions in real time using its camera and microphone. Input is the user's facial expressions and voice data, while output is emotional state information. Specifically, it runs an emotion recognition algorithm to classify the emotional state.

[0199] Step 7:

[0200] The server adjusts notification content based on emotional state information. The input is the recognized emotional state, and the output is a notification message adjusted according to that emotional state. Specifically, if the user is stressed, the server generates a message suggesting ways to relax.

[0201] Step 8:

[0202] The server stores the user's meal history and emotional data through an information storage system. Inputs are the amount of salt consumed and emotional data for each meal, and output is the accumulated dataset. Specifically, it writes data to a database in preparation for long-term analysis.

[0203] Step 9:

[0204] The server analyzes the accumulated data using information analysis tools to extract temporal trends in the user's health status. The input is the accumulated dataset, and the output is the analysis results regarding the temporal changes in the user's salt intake and emotional state. Specifically, it runs a data analysis algorithm and generates a report.

[0205] (Application Example 2)

[0206] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0207] Excessive salt intake in the diet of the elderly is highly likely to have adverse effects on their health, and their mental state is also an important factor in health management. However, conventional dietary management systems have difficulty simultaneously managing salt intake and emotional state, and a more comprehensive health management system is needed. Furthermore, providing this information to family members living remotely is expected to provide better support.

[0208] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0209] In this invention, the server includes a digital image analysis means for receiving and analyzing images of photographed meals, a nutrition calculation means for identifying food from the analyzed images and retrieving and summing its salt content from a database, and an emotion recognition means for recognizing the user's emotional state and reflecting this in notifications and warnings regarding salt intake. This enables more comprehensive health management by simultaneously managing salt intake in meals and the emotional state of elderly people, and by providing information to family members in remote locations.

[0210] "Digital image analysis means" refers to technology that receives images of food, analyzes their contents, and identifies the food items.

[0211] The "nutrition calculation method" is a technique that retrieves the amount of salt in foods identified by image analysis from a database and sums it up.

[0212] "Information generation means" refers to a technology that generates and transmits information to a notification device based on the calculated amount of salt.

[0213] An "alert issuance system" is a technology that issues a warning to users and relevant parties when a salt level exceeding the standard is detected.

[0214] "Information storage means" refers to technology that stores analysis results and calculation results as data and manages data on eating habits over a long period of time.

[0215] "Emotion recognition means" refers to technology that recognizes the user's emotional state and reflects it in notifications and warnings regarding salt intake.

[0216] "Family notification means" refers to technology that notifies relatives in remote locations of meal and emotional data, making it possible for them to review and verify it.

[0217] "Data analysis means" refers to technology that enables analysis based on the user's salt intake and emotional state over time.

[0218] This invention is designed as a health management system that simultaneously considers the salt intake management and emotional state of elderly individuals. Users can use a smartphone or tablet device to take photos of each meal and send the images to the server in real time.

[0219] The server uses tools such as "OpenCV" and "TensorFlow" as image analysis tools to identify food items from received images of meals. Based on the food information obtained through this analysis, it retrieves the amount of salt for each food item from an "SQL database" as a nutrition calculation tool and sums them up. The resulting salt amount is then notified in real time via a notification generation tool to the user's smartphone application or the devices of registered relatives.

[0220] In addition, video and audio captured through the user's camera or microphone are analyzed using emotion recognition technologies such as "Google Cloud Vision" and "Microsoft® Azure® Cognitive Services" to determine the user's emotional state. Notifications incorporating this emotional data provide personalized advice, including suggestions for relaxation methods if the user is feeling stressed.

[0221] The data storage method involves accumulating user salt intake and emotional data over a long period using a NoSQL database, among other methods. This data is then used by data analysis tools to analyze the correlation between salt intake and emotions over time.

[0222] For example, if a user chooses a dish high in salt for lunch, an image of it is sent to the system, and the amount of salt is immediately calculated. If the system detects that the user is experiencing stress during the meal, a notification is sent through the application to the user and their family members, such as, "This dish is high in salt. How about adding some vegetables? We have prepared some calming music for you to help you relax."

[0223] This system enables comprehensive management of users' health status, contributing to an improved quality of life for the elderly. Furthermore, it accepts user input in the form of prompts to be input into the generating AI model, such as, "Please generate suggestions if the meal Grandma is about to eat is high in salt. Also, please consider relaxation measures if she appears anxious."

[0224] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0225] Step 1:

[0226] The user takes a picture of their meal with their smartphone camera. The captured image is saved on the device and sent to the server. The input for this step is the image of the meal, and the output is the image data sent to the server.

[0227] Step 2:

[0228] The server analyzes the received image data using "OpenCV" to identify the food items within the image. This process takes the image data as input to determine the type of food item, which is then output. The identified information is then sent to the next processing step.

[0229] Step 3:

[0230] The server retrieves salt content information for each food item identified from the SQL database. The input is the identified food item information, and the output is the salt content data corresponding to each food item. This allows for an understanding of the overall nutritional components of each food item.

[0231] Step 4:

[0232] The salt content of each identified food item is summed up to calculate the total salt content. The input for this step is the salt content data for each food item, and the output is the total salt content. The server performs this calculation, providing the necessary basic data for notifying the user.

[0233] Step 5:

[0234] The server uses "Google Cloud Vision" to analyze video and audio acquired from the camera or microphone to recognize the user's emotional state. This process uses the user's video and audio input as the data source and outputs emotional state information.

[0235] Step 6:

[0236] The server uses a notification generation mechanism to create and send notifications to the user and registered relatives based on total salt intake and emotional state. The input is salt intake and emotional state, and the output is a pre-configured notification message. This process delivers notifications, including advice tailored to the user's purpose.

[0237] Step 7:

[0238] The notified data is stored in a NoSQL database and accumulated as a history of salt intake and emotional state. Inputs are dietary and emotional data, and the accumulated information is output. This enables long-term health management for the user.

[0239] Step 8:

[0240] The server uses data analysis tools to analyze accumulated data and interpret changes in salt intake and emotions over time. The input is accumulated historical data, and the output is the interpreted analysis results. This will provide insights that can be useful for future health management.

[0241] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0242] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0243] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0244] [Second Embodiment]

[0245] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0246] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0247] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0248] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0249] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0250] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0251] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0252] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0253] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0254] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0255] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0256] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0257] The system according to the present invention helps elderly people living alone to easily manage their salt intake, and a specific embodiment thereof will be described below.

[0258] The system starts when the user takes a picture of their meal with a smartphone or dedicated device. The captured image is saved on the device and sent to a server via the network. Upon receiving the image, the server uses image analysis to identify the food items. A deep learning model is used here to label the different food items in the image.

[0259] For each identified food item, the server retrieves information on the corresponding salt content from its internal database and uses a salt content calculation device to sum these values ​​and calculate the total salt content. This calculation result is then used by a notification generation device to generate a notification message to be sent to the user and their relatives who are remotely connected. This message includes the salt content for each food item and the total salt content.

[0260] If the salt intake exceeds a pre-set standard, the server sends a signal to the terminal via a warning system, instructing the terminal to sound a warning. Furthermore, the analysis and calculation results are stored in the server's database as the user's meal history by a data storage system.

[0261] Furthermore, by analyzing this accumulated data using data analysis tools, it becomes possible to help users manage their health in the long term. This allows users to easily understand the temporal trends in their salt intake, and relatives can remotely provide dietary advice.

[0262] As described above, the present invention allows users to easily manage the amount of salt they consume from their daily meals, and furthermore, enables relatives to provide appropriate advice based on that information. This makes it possible to provide a system that supports the maintenance of the health of the elderly.

[0263] The following describes the processing flow.

[0264] Step 1:

[0265] Users take photos of their meals using their smartphones or dedicated devices. The captured images are automatically saved to the device.

[0266] Step 2:

[0267] The device uploads the saved photos to a designated cloud server. The photo data is transferred using an internet connection during the upload process.

[0268] Step 3:

[0269] The server activates an image analysis system to analyze the received photo data. This process uses deep learning technology to identify each food item in the image.

[0270] Step 4:

[0271] The server retrieves salt content information for each identified food item from its internal database. It aggregates the salt content data for each food item and calculates the total salt content using a salt content calculation device.

[0272] Step 5:

[0273] The server uses the calculated total salt content and the salt content information of individual foods to generate a notification message. This notification message is created for the user and their relatives.

[0274] Step 6:

[0275] The server sends the generated notification message to the user's and their relatives' devices via a notification generation mechanism. Registered contact information is used for sending the message.

[0276] Step 7:

[0277] If the salt content exceeds a set standard, the server sends an instruction to the terminal using a warning system, and the terminal notifies the user by sounding a warning.

[0278] Step 8:

[0279] The server will save the analysis and calculation results in a data storage system. This data will be stored as the user's long-term dietary history.

[0280] Step 9:

[0281] The server activates the data analysis means using the accumulated data and analyzes the salt intake based on the passage of time. The analysis results are provided to the user and relatives on a regular basis.

[0282] (Example 1)

[0283] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0284] The impact of diet on health is significant. Especially for the elderly, excessive salt intake can be a factor increasing health risks. However, for the elderly who are increasingly eating alone, it is a difficult problem to appropriately manage their daily salt intake. Also, even if relatives living far away want to support the daily diet, the physical distance often becomes an obstacle. In response to this, there is a demand for providing a system that allows the elderly to easily and effectively manage their diet and enables relatives living remotely to also provide support.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following respective means.

[0286] In this invention, the server includes an analysis device means for receiving a meal image captured from a location information terminal and detecting food components by digital image processing, a nutrition analysis device means for extracting the amount of each component of the meal item from the data storage device based on the detected meal item and calculating the total value thereof, and an information notification device means for transmitting the calculated total component amount to the user via a communication device. Thereby, the user can easily manage the daily salt intake of their meals, and relatives can also effectively support the diet remotely.

[0287] The "location information terminal" is an electronic device for capturing images of meals and other contents and transmitting them to the server.

[0288] "Digital image processing" refers to a technical method that analyzes received image data to detect food components.

[0289] "Food components" refer to the individual nutritional elements and their attributes contained in a meal.

[0290] An "analysis device" is a general term for hardware or software that analyzes the components and characteristics of food based on received data.

[0291] A "nutrition analyzer" is a device that calculates the amount of each component as a numerical value based on the detected food components and then calculates the sum of those values.

[0292] A "data storage device" refers to a database that stores information related to food ingredients and allows for searching and retrieving this information as needed.

[0293] An "information notification device" is a means of presenting information on the calculated sum of component amounts to the user, and functions as a communication device.

[0294] A "signal issuing device" is a device that generates alarms based on set criteria values ​​to alert the user.

[0295] An "information storage device" is a device that stores the results of analysis and calculations over a long period of time and manages historical data on food intake.

[0296] A "remote communication device" is a device or system that can transmit captured image information to an individual or group located at a distance.

[0297] A "data processing device" refers to a device that analyzes accumulated information and generates detailed reports based on changes over time.

[0298] This invention provides a system that allows elderly people living alone to easily manage their daily salt intake. The system starts when the user uses a location-based terminal to take a picture of their meal, saves the image to the terminal, and transmits it to a server via a network. The server analyzes the received image using digital image processing to detect food components. Various deep learning models specialized in image analysis are used for this purpose.

[0299] The server extracts component amounts from data storage based on the components obtained by the analysis device. These component amounts are then summed up using a nutritional analyzer to calculate the total salt intake. The calculated results are reported to the user via an information notification device. The notification includes the component amounts for each food item and the total component amounts, allowing the user to accurately control their intake.

[0300] If a user's salt intake exceeds a set limit, the server issues an alarm via a signaling device. Upon receiving the alarm, the terminal emits an audible or visual warning to alert the user. Analysis and calculation results are stored in an information storage device, and based on this historical data, the server uses a data processing device to analyze long-term nutritional intake trends and provide advice for maintaining health.

[0301] Furthermore, the captured image information can be shared with relatives in distant locations via a remote communication device, allowing them to provide direct support remotely. This sharing of information enables relatives to understand the user's daily eating habits and provide appropriate feedback.

[0302] A concrete example is a scenario where a user eats ramen and salad for lunch and takes a picture of it with their device. The server analyzes this image, identifies the salt content of each food item, and notifies the user of the results. The prompt used for the generating AI model is, "Take pictures of your ramen and salad for lunch and calculate the salt content of each." Through this prompt, the server can perform appropriate image analysis and provide information.

[0303] The flow of the specific process in Example 1 will be described with reference to FIG. 11.

[0304] Step 1:

[0305] The user takes a photo of a meal using a smartphone or a dedicated photography device. The captured image is saved in the terminal. This image data serves as the input for the next process. As a specific operation, the camera app on the terminal is launched, the meal is framed, and the shutter is pressed.

[0306] Step 2:

[0307] The terminal sends the saved image to the server via the network. Here, the input is the image file saved in the terminal, and the output is the image data received by the server. The terminal uses its communication function to upload the image file to the server according to an appropriate communication protocol.

[0308] Step 3:

[0309] The server analyzes the received image data using digital image processing technology. In this step, the image data transferred to the server is used as the input, and label data identifying the food is generated as the output. The server uses a generated AI model to extract the characteristics of the food and perform labeling.

[0310] Step 4:

[0311] Based on the analyzed label data, the server searches the data storage device to obtain the salt content of each food. The input is the food label data, and the output is the corresponding salt content information. The server queries the database using the name of the identified food as the key and obtains the corresponding salt content.

[0312] Step 5:

[0313] The server calculates the total salt content by summing up the salt content obtained using a nutritional analyzer. In this step, the salt content information of each food item is used as input, and the output is the calculated total salt content. Specifically, the process involves summing up the total salt content using numerical calculations.

[0314] Step 6:

[0315] The server notifies the user via an information notification device based on the total salt content result. The input is the calculated total salt content, and the output is a notification message displayed on the user's terminal. The server generates the notification content and sends the message to the user.

[0316] Step 7:

[0317] The server determines whether the calculated total salt content exceeds the set standard, and if so, issues an alarm via a signaling device. The inputs are the total salt content and the set standard, and the output is a warning signal. The server compares the value to the standard and, if necessary, generates a warning signal and sends it to the terminal.

[0318] Step 8:

[0319] The server stores analysis and calculation results in an information storage device and records them as long-term food intake history data. The input is a series of analysis and calculation results, and the output is historical data stored in a database. The server stores the information through a data management system.

[0320] (Application Example 1)

[0321] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0322] Managing salt intake in meals smoothly for elderly people living alone is a challenge. Existing methods make it difficult to effectively utilize information even when photos of meals are taken, requiring continuous support from family members or caregivers. Furthermore, the inability to visually manage salt intake poses a risk of problems. Therefore, there is a need for a system that allows users to easily understand their salt intake and manage their health while sharing information with relatives.

[0323] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0324] In this invention, the server includes image analysis means for receiving and analyzing images of photographed meals, visualization means for displaying the user's long-term dietary salt intake in a chart format, and remote access means for enabling information sharing among multiple users. This makes it possible for users to easily manage their salt intake and manage their health while sharing information with relatives.

[0325] "Image analysis means" refers to a process used to receive images of a photographed meal and identify identifiable food items.

[0326] The "salt content calculation method" is a process that retrieves the salt content of food items identified based on the analyzed images from a database and then sums up the total salt content.

[0327] The "notification generation method" is a procedure for notifying the user's device of the calculated amount of salt.

[0328] A "warning signaling mechanism" is a system that issues a warning signal when the salt content exceeds the standard.

[0329] A "data storage method" is a process for storing analysis results and calculation results over a long period of time so that they can be analyzed later.

[0330] "Visualization means" refers to a function that displays the user's long-term salt intake data in a chart format.

[0331] "Remote access means" refers to technologies that enable multiple users to share and access information via the internet.

[0332] "Family notification method" refers to a method of notifying the user's relatives of the photographed meal images so that they can view them remotely.

[0333] "Data analysis tools" refer to tools used to analyze users' salt intake over time.

[0334] This invention provides a system that allows elderly people to easily manage their salt intake through meals. The system begins when a user takes a picture of their meal using a smartphone or other device. The device sends the captured image to a server, which then performs image analysis on the received image. The image analysis utilizes a deep learning model, enabling automatic identification of food items. For each identified food item, the server retrieves the salt content from a database and calculates the total salt intake.

[0335] The calculation results are notified to the user's device and to relatives located remotely. The notification includes information on the amount of salt in each food item and the total amount of salt. If the salt intake exceeds a pre-set standard, a signal is sent to the device to emit a warning sound. Furthermore, the server stores the analysis and calculation results as data for use in long-term health management. This stored data is visualized in various charts and graphs, making it easy to check salt intake over time. This also allows relatives to provide dietary advice remotely.

[0336] As a concrete example, by using a prompt such as, "Take a picture of your lunch, is today's salt intake okay?", users can easily enjoy the benefits of the invention. This system uses software such as Python and TensorFlow to process and calculate data. Smartphones and tablets are commonly used as hardware. In this way, the present invention provides comprehensive support for easily managing the amount of salt consumed from daily meals.

[0337] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0338] Step 1:

[0339] A user takes a picture of their meal using their smartphone camera. The captured image is saved on the device and then sent to a server via the internet. In this process, the image is the input, and the data transmission to the server is the output.

[0340] Step 2:

[0341] The server analyzes the received images using a deep learning model. Specifically, each food item in the image is identified. A list of labeled food items is output from the image data. An object recognition algorithm is used in this process.

[0342] Step 3:

[0343] The server retrieves the salt content of each food item from the database based on the label. It takes a list of foods as input and outputs the corresponding salt content information. Data retrieval is performed here using SQL queries, etc.

[0344] Step 4:

[0345] The salt content calculation method calculates the total salt content by summing up the acquired salt amounts. An array containing the individual salt amounts is taken as input, and the total salt content is output. The calculation is performed using standard addition.

[0346] Step 5:

[0347] The calculated total salt intake is sent as a notification to the user's device and their relatives' devices. The notification message is output using the total salt intake data as input. A messaging API is used for this process.

[0348] Step 6:

[0349] The server stores the calculation results in a database, accumulating them as a long-term dietary history. Here, the analysis results and calculation results serve as input, and the accumulated data records are output. A database management system is used for data storage.

[0350] Step 7:

[0351] The system visualizes accumulated data in graphs and charts, allowing users to review their past salt intake history. Historical data retrieved from the database is used as input, and visualized information is output. Data analysis software is used for visualization.

[0352] Step 8:

[0353] If the user-set threshold is exceeded, a signal is sent to the device to emit a warning sound. The total salt content is used as input to output the warning sound. This process utilizes the device's notification system.

[0354] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0355] This invention aims to more effectively manage the health of the elderly by combining a salt management system for meals with a user emotion recognition function. Specific embodiments are described below.

[0356] The system initiates the process by having the user take a picture of their meal with their smartphone or a dedicated device. The captured image is saved on the device and uploaded to the server. The server uses image analysis to identify the food in the received image and retrieves salt content information from a database for each recognized food item.

[0357] Subsequently, the server uses a salt content calculation device to sum up the identified salt amounts and calculate the total salt content. The calculation result is sent as a message to the user and their relatives by a notification generation device. This message includes the salt content of individual foods and an indicator tailored to the newly adopted emotion.

[0358] The emotion engine, a key feature of this invention, has the ability to recognize the user's emotions in real time. The user's emotions are captured through camera or voice input and analyzed by an emotion recognition algorithm. Based on the results of this emotion analysis, notifications and warnings regarding salt intake are incorporated. For example, if the user is feeling stressed, the notification is also sent to relatives, including advice that takes stress reduction into consideration.

[0359] Furthermore, emotional data is stored in a data storage system along with the user's dietary history. This data is used by data analysis tools to analyze the user's salt intake over time, supporting long-term health management. In addition, fluctuations based on the user's emotional data are recorded over a long period and provided to family members. This enables comprehensive health management, including the user's psychological state.

[0360] For example, if a user feels anxious during a meal, the system recognizes this through an emotion engine. The server analyzes this and sends advice to the family, along with a notification about the amount of salt consumed, suggesting ways to relax and alleviate the anxiety. Based on this, the family can provide appropriate support to the user.

[0361] Thus, the present invention provides a system that supports a healthier lifestyle for the elderly by simultaneously managing salt intake and the user's emotions.

[0362] The following describes the processing flow.

[0363] Step 1:

[0364] The user takes a photo of their meal using their smartphone or a dedicated device. This triggers the system to start operating.

[0365] Step 2:

[0366] The device temporarily stores the captured images and uploads them to the server via the network. The device is configured to automatically upload images after they are captured.

[0367] Step 3:

[0368] The server analyzes the received images based on image analysis tools to identify the food items in the photographs. AI-based image recognition technology is used for food identification.

[0369] Step 4:

[0370] The server retrieves the salt content of each food item from a database based on the data of the identified food item. This makes it possible to collect accurate salt information.

[0371] Step 5:

[0372] The server uses a salt content calculation device to add up the salt content of each food item and calculate the total salt content. This value will be used in subsequent notifications.

[0373] Step 6:

[0374] To recognize the user's emotions in real time, the device uses a camera or microphone to acquire emotional data from the user's facial expressions and voice.

[0375] Step 7:

[0376] Emotional data acquired from the terminal is sent to a server, where an emotion engine processes it and determines the user's emotions.

[0377] Step 8:

[0378] The server combines salt levels and emotional data to generate a notification message. This message includes advice based on the salt level and the corresponding emotional state.

[0379] Step 9:

[0380] The server sends the generated notifications to the user's and registered relatives' devices. It provides not only salt intake calculation results but also emotionally-driven advice.

[0381] Step 10:

[0382] If the server detects a salt level exceeding the standard or any resulting emotional changes, it will use a warning system on the terminal to emit a warning sound or message.

[0383] Step 11:

[0384] The server stores analysis results and emotional data in a data storage system. This data is used for long-term health management and tracking changes in the user's emotional state.

[0385] Step 12:

[0386] The stored data is analyzed using data analysis tools. This makes it easier to understand the user's health status, salt intake, and emotional state trends.

[0387] This system allows for the simultaneous management of the dietary habits and emotional well-being of elderly individuals, enabling more comprehensive health management.

[0388] (Example 2)

[0389] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0390] For elderly individuals, managing salt intake is crucial for maintaining health. However, accurately determining the salt content of meals is generally difficult. Furthermore, the impact of emotional state during meals on health should also be considered. Conventional systems have not comprehensively considered both salt management and the user's emotional state. Therefore, there is a need for technology that can simultaneously achieve appropriate management of dietary content and care for emotional state.

[0391] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0392] In this invention, the server includes an image processing means for receiving and analyzing images of photographed meals, a quantity calculation means for identifying food from the analyzed images and obtaining and summing the salt content from an information storage device, and an emotion analysis means for recognizing the user's emotions and reflecting them in the analysis results and notifications. This enables comprehensive health management that accurately manages the amount of salt in the user's meals while also taking into account their emotional state.

[0393] "Image processing means" refers to technology that provides functions for receiving images of photographed food and analyzing those images.

[0394] The "quantity calculation means" is a technology for obtaining and summing the salt content of food identified from the analyzed images by retrieving it from an information storage device.

[0395] "Notification generation means" refers to a technology equipped with a function for transmitting calculated salt content information to a notification device.

[0396] A "warning issuance method" is a technology that provides a method for issuing a warning when the salt content exceeds the standard.

[0397] "Information storage means" refers to technology for accumulating analysis results and calculation results to store dietary information over a long period of time.

[0398] "Emotion analysis means" refers to technology that recognizes the user's emotions and reflects them in the analysis results and notifications.

[0399] "Notification adjustment means" refers to technology that has a function to adjust notification content based on the user's emotional state.

[0400] "Family notification means" refers to a technology that notifies a remote relative of a captured image, allowing that relative to view it.

[0401] "Information analysis means" refers to technology for performing analysis based on the amount of salt consumed by the user and the time course of their emotional state.

[0402] This invention is a system that supports health management by comprehensively analyzing a user's dietary management and emotional state. Specific embodiments are described below.

[0403] Users take photos of their meals using their smartphones or dedicated terminals. These terminals store the images and upload them to a server via the network. The server analyzes the received images using image processing software (e.g., OpenCV or TensorFlow) to identify the food items. Machine learning algorithms are used in this process, based on the food's shape, color, patterns, and other characteristics.

[0404] The server calculates the salt content using information about the identified food items. This information is obtained from a nutritional database. The server then sends the total salt content to the user and distant relatives via a notification generation system. This notification includes details of the meal and the total salt content.

[0405] Furthermore, the user's emotions are captured using the device's camera and microphone. These are processed in real time by emotion analysis algorithms (e.g., Amazon Rekognition or Google Cloud Speech-to-Text). Based on the recognized emotions, the server adjusts the notification content and sends it to the relative, including appropriate advice.

[0406] This system stores users' dietary history and emotional data using information storage mechanisms for long-term use. The data is analyzed using information analysis mechanisms to track changes in the user's salt intake and emotional state over time. This enables comprehensive health management.

[0407] For example, if a user feels down while eating a fish dish at dinner, the system recognizes this situation and sends a notification to a relative that includes advice on how to improve their mood, along with information on the amount of salt consumed.

[0408] An example of a prompt to input into the generative AI model is as follows: "I would like to detail the specific processes of a health management system for the elderly. This includes everything from taking photos of meals to managing salt intake, and also mentions the role of sentiment analysis."

[0409] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0410] Step 1:

[0411] The user takes a picture of their meal using a smartphone or dedicated device. The captured image is saved to the device. The input is image data generated by the user, and the output is an image file stored on the device. Specifically, the camera application is used to capture the image and save it to the device's storage.

[0412] Step 2:

[0413] The terminal uploads the saved image to the server via the network. The input is the image file saved in step 1, and the output is the image data transferred to the server. Specifically, the image file is sent to a specified endpoint on the server via the internet connection.

[0414] Step 3:

[0415] The server analyzes the received images through image processing tools. It uses software such as OpenCV and TensorFlow to identify food items within the images. The input is image data uploaded to the server, and the output is identification information for the food items in the images. Specifically, it runs a machine learning model to analyze shape, color, and pattern to identify the food items.

[0416] Step 4:

[0417] The server retrieves the salt content of identified foods from a database and sums them up. The input is a list of identified foods, and the output is the salt content of each food and the sum of those salt content values. Specifically, it sends a query to the database for each food and adds up the retrieved salt content data.

[0418] Step 5:

[0419] The server generates a message containing the total salt content and food details, and sends it to the user and their relatives. The input is the calculated total salt content and food details, and the output is the notification message. Specifically, it uses a notification generation engine to create and send emails and app notifications.

[0420] Step 6:

[0421] The device recognizes the user's emotions in real time using its camera and microphone. Input is the user's facial expressions and voice data, while output is emotional state information. Specifically, it runs an emotion recognition algorithm to classify the emotional state.

[0422] Step 7:

[0423] The server adjusts notification content based on emotional state information. The input is the recognized emotional state, and the output is a notification message adjusted according to that emotional state. Specifically, if the user is stressed, the server generates a message suggesting ways to relax.

[0424] Step 8:

[0425] The server stores the user's meal history and emotional data through an information storage system. Inputs are the amount of salt consumed and emotional data for each meal, and output is the accumulated dataset. Specifically, it writes data to a database in preparation for long-term analysis.

[0426] Step 9:

[0427] The server analyzes the accumulated data using information analysis tools to extract temporal trends in the user's health status. The input is the accumulated dataset, and the output is the analysis results regarding the temporal changes in the user's salt intake and emotional state. Specifically, it runs a data analysis algorithm and generates a report.

[0428] (Application Example 2)

[0429] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0430] Excessive salt intake in the diet of the elderly is highly likely to have adverse effects on their health, and their mental state is also an important factor in health management. However, conventional dietary management systems have difficulty simultaneously managing salt intake and emotional state, and a more comprehensive health management system is needed. Furthermore, providing this information to family members living remotely is expected to provide better support.

[0431] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0432] In this invention, the server includes a digital image analysis means for receiving and analyzing images of photographed meals, a nutrition calculation means for identifying food from the analyzed images and retrieving and summing its salt content from a database, and an emotion recognition means for recognizing the user's emotional state and reflecting this in notifications and warnings regarding salt intake. This enables more comprehensive health management by simultaneously managing salt intake in meals and the emotional state of elderly people, and by providing information to family members in remote locations.

[0433] "Digital image analysis means" refers to technology that receives images of food, analyzes their contents, and identifies the food items.

[0434] The "nutrition calculation method" is a technique that retrieves the amount of salt in foods identified by image analysis from a database and sums it up.

[0435] "Information generation means" refers to a technology that generates and transmits information to a notification device based on the calculated amount of salt.

[0436] An "alert issuance system" is a technology that issues a warning to users and relevant parties when a salt level exceeding the standard is detected.

[0437] "Information storage means" refers to technology that stores analysis results and calculation results as data and manages data on eating habits over a long period of time.

[0438] "Emotion recognition means" refers to technology that recognizes the user's emotional state and reflects it in notifications and warnings regarding salt intake.

[0439] "Family notification means" refers to technology that notifies relatives in remote locations of meal and emotional data, making it possible for them to review and verify it.

[0440] "Data analysis means" refers to technology that enables analysis based on the user's salt intake and emotional state over time.

[0441] This invention is designed as a health management system that simultaneously considers the salt intake management and emotional state of elderly individuals. Users can use a smartphone or tablet device to take photos of each meal and send the images to the server in real time.

[0442] The server uses tools such as "OpenCV" and "TensorFlow" as image analysis tools to identify food items from received images of meals. Based on the food information obtained through this analysis, it retrieves the amount of salt for each food item from an "SQL database" as a nutrition calculation tool and sums them up. The resulting salt amount is then notified in real time via a notification generation tool to the user's smartphone application or the devices of registered relatives.

[0443] In addition, video and audio captured through the user's camera or microphone are analyzed using emotion recognition technologies such as "Google Cloud Vision" and "Microsoft Azure Cognitive Services" to determine the user's emotional state. Notifications incorporating this emotional data provide personalized advice, including suggestions for relaxation methods if the user is feeling stressed.

[0444] The data storage method involves accumulating user salt intake and emotional data over a long period using a NoSQL database, among other methods. This data is then used by data analysis tools to analyze the correlation between salt intake and emotions over time.

[0445] For example, if a user chooses a dish high in salt for lunch, an image of it is sent to the system, and the amount of salt is immediately calculated. If the system detects that the user is experiencing stress during the meal, a notification is sent through the application to the user and their family members, such as, "This dish is high in salt. How about adding some vegetables? We have prepared some calming music for you to help you relax."

[0446] This system enables comprehensive management of users' health status, contributing to an improved quality of life for the elderly. Furthermore, it accepts user input in the form of prompts to be input into the generating AI model, such as, "Please generate suggestions if the meal Grandma is about to eat is high in salt. Also, please consider relaxation measures if she appears anxious."

[0447] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0448] Step 1:

[0449] The user takes a picture of their meal with their smartphone camera. The captured image is saved on the device and sent to the server. The input for this step is the image of the meal, and the output is the image data sent to the server.

[0450] Step 2:

[0451] The server analyzes the received image data using "OpenCV" to identify the food items within the image. This process takes the image data as input to determine the type of food item, which is then output. The identified information is then sent to the next processing step.

[0452] Step 3:

[0453] The server retrieves salt content information for each food item identified from the SQL database. The input is the identified food item information, and the output is the salt content data corresponding to each food item. This allows for an understanding of the overall nutritional components of each food item.

[0454] Step 4:

[0455] The salt content of each identified food item is summed up to calculate the total salt content. The input for this step is the salt content data for each food item, and the output is the total salt content. The server performs this calculation, providing the necessary basic data for notifying the user.

[0456] Step 5:

[0457] The server uses "Google Cloud Vision" to analyze video and audio acquired from the camera or microphone to recognize the user's emotional state. This process uses the user's video and audio input as the data source and outputs emotional state information.

[0458] Step 6:

[0459] The server uses a notification generation mechanism to create and send notifications to the user and registered relatives based on total salt intake and emotional state. The input is salt intake and emotional state, and the output is a pre-configured notification message. This process delivers notifications, including advice tailored to the user's purpose.

[0460] Step 7:

[0461] The notified data is stored in a NoSQL database and accumulated as a history of salt intake and emotional state. Inputs are dietary and emotional data, and the accumulated information is output. This enables long-term health management for the user.

[0462] Step 8:

[0463] The server uses data analysis tools to analyze accumulated data and interpret changes in salt intake and emotions over time. The input is accumulated historical data, and the output is the interpreted analysis results. This will provide insights that can be useful for future health management.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0465] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0467] [Third Embodiment]

[0468] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0469] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0470] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0472] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0474] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0475] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0476] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0479] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0480] The system according to the present invention helps elderly people living alone to easily manage their salt intake, and a specific embodiment thereof will be described below.

[0481] The system starts when the user takes a picture of their meal with a smartphone or dedicated device. The captured image is saved on the device and sent to a server via the network. Upon receiving the image, the server uses image analysis to identify the food items. A deep learning model is used here to label the different food items in the image.

[0482] For each identified food item, the server retrieves information on the corresponding salt content from its internal database and uses a salt content calculation device to sum these values ​​and calculate the total salt content. This calculation result is then used by a notification generation device to generate a notification message to be sent to the user and their relatives who are remotely connected. This message includes the salt content for each food item and the total salt content.

[0483] If the salt intake exceeds a pre-set standard, the server sends a signal to the terminal via a warning system, instructing the terminal to sound a warning. Furthermore, the analysis and calculation results are stored in the server's database as the user's meal history by a data storage system.

[0484] Furthermore, by analyzing this accumulated data using data analysis tools, it becomes possible to help users manage their health in the long term. This allows users to easily understand the temporal trends in their salt intake, and relatives can remotely provide dietary advice.

[0485] As described above, the present invention allows users to easily manage the amount of salt they consume from their daily meals, and furthermore, enables relatives to provide appropriate advice based on that information. This makes it possible to provide a system that supports the maintenance of the health of the elderly.

[0486] The following describes the processing flow.

[0487] Step 1:

[0488] Users take photos of their meals using their smartphones or dedicated devices. The captured images are automatically saved to the device.

[0489] Step 2:

[0490] The device uploads the saved photos to a designated cloud server. The photo data is transferred using an internet connection during the upload process.

[0491] Step 3:

[0492] The server activates an image analysis system to analyze the received photo data. This process uses deep learning technology to identify each food item in the image.

[0493] Step 4:

[0494] The server retrieves salt content information for each identified food item from its internal database. It aggregates the salt content data for each food item and calculates the total salt content using a salt content calculation device.

[0495] Step 5:

[0496] The server uses the calculated total salt content and the salt content information of individual foods to generate a notification message. This notification message is created for the user and their relatives.

[0497] Step 6:

[0498] The server sends the generated notification message to the user's and their relatives' devices via a notification generation mechanism. Registered contact information is used for sending the message.

[0499] Step 7:

[0500] If the salt content exceeds a set standard, the server sends an instruction to the terminal using a warning system, and the terminal notifies the user by sounding a warning.

[0501] Step 8:

[0502] The server will save the analysis and calculation results in a data storage system. This data will be stored as the user's long-term dietary history.

[0503] Step 9:

[0504] The server uses the accumulated data to activate data analysis tools and perform an analysis of salt intake over time. The results of this analysis are provided periodically to the user and their relatives.

[0505] (Example 1)

[0506] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0507] Diet has a significant impact on health, and excessive salt intake, in particular, can increase health risks for the elderly. However, for the elderly, who increasingly eat alone, managing their daily salt intake appropriately is a difficult problem. Furthermore, even if relatives living far away want to support their daily diet, physical distance is often an obstacle. To address this, there is a need for a system that allows the elderly to easily and effectively manage their diet, and that enables relatives living far away to provide support.

[0508] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0509] In this invention, the server includes an analysis device means that receives a meal image captured from a location information terminal and detects food components by digital image processing; a nutritional analysis device means that extracts the amount of each component of the meal item from a data storage device based on the detected meal item and calculates its sum; and an information notification device means that transmits the calculated sum of component amounts to the user via a communication device. This allows the user to easily manage their daily salt intake, and enables relatives to remotely provide effective support for their diet.

[0510] A "location information terminal" is an electronic device used to capture images of food or other content and transmit them to a server.

[0511] "Digital image processing" refers to a technical method that analyzes received image data to detect food components.

[0512] "Food components" refer to the individual nutritional elements and their attributes contained in a meal.

[0513] An "analysis device" is a general term for hardware or software that analyzes the components and characteristics of food based on received data.

[0514] A "nutrition analyzer" is a device that calculates the amount of each component as a numerical value based on the detected food components and then calculates the sum of those values.

[0515] A "data storage device" refers to a database that stores information related to food ingredients and allows for searching and retrieving this information as needed.

[0516] An "information notification device" is a means of presenting information on the calculated sum of component amounts to the user, and functions as a communication device.

[0517] A "signal issuing device" is a device that generates alarms based on set criteria values ​​to alert the user.

[0518] An "information storage device" is a device that stores the results of analysis and calculations over a long period of time and manages historical data on food intake.

[0519] A "remote communication device" is a device or system that can transmit captured image information to an individual or group located at a distance.

[0520] A "data processing device" refers to a device that analyzes accumulated information and generates detailed reports based on changes over time.

[0521] This invention provides a system that allows elderly people living alone to easily manage their daily salt intake. The system starts when the user uses a location-based terminal to take a picture of their meal, saves the image to the terminal, and transmits it to a server via a network. The server analyzes the received image using digital image processing to detect food components. Various deep learning models specialized in image analysis are used for this purpose.

[0522] The server extracts component amounts from data storage based on the components obtained by the analysis device. These component amounts are then summed up using a nutritional analyzer to calculate the total salt intake. The calculated results are reported to the user via an information notification device. The notification includes the component amounts for each food item and the total component amounts, allowing the user to accurately control their intake.

[0523] If a user's salt intake exceeds a set limit, the server issues an alarm via a signaling device. Upon receiving the alarm, the terminal emits an audible or visual warning to alert the user. Analysis and calculation results are stored in an information storage device, and based on this historical data, the server uses a data processing device to analyze long-term nutritional intake trends and provide advice for maintaining health.

[0524] Furthermore, the captured image information can be shared with relatives in distant locations via a remote communication device, allowing them to provide direct support remotely. This sharing of information enables relatives to understand the user's daily eating habits and provide appropriate feedback.

[0525] A concrete example is a scenario where a user eats ramen and salad for lunch and takes a picture of it with their device. The server analyzes this image, identifies the salt content of each food item, and notifies the user of the results. The prompt used for the generating AI model is, "Take pictures of your ramen and salad for lunch and calculate the salt content of each." Through this prompt, the server can perform appropriate image analysis and provide information.

[0526] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0527] Step 1:

[0528] The user takes a picture of their meal using a smartphone or a dedicated camera device. The captured image is saved to the device. This image data becomes the input for the next process. Specifically, the user launches the camera app on their device, frames the meal, and presses the shutter button.

[0529] Step 2:

[0530] The terminal sends the saved image to the server over the network. Here, the input is the image file stored on the terminal, and the output is the image data received by the server. The terminal uses its communication function to upload the image file to the server according to the appropriate communication protocol.

[0531] Step 3:

[0532] The server analyzes the received image data using digital image processing technology. In this step, the image data transferred to the server is used as input, and label data identifying the food is generated as output. The server uses a generative AI model to extract the features of the food and label it.

[0533] Step 4:

[0534] The server searches the data storage device based on the analyzed label data to obtain the salt content of each food item. The input is food label data, and the output is the corresponding salt content information. The server queries the database using the name of the identified food item as a key to retrieve the corresponding salt content.

[0535] Step 5:

[0536] The server calculates the total salt content by summing up the salt content obtained using a nutritional analyzer. In this step, the salt content information of each food item is used as input, and the output is the calculated total salt content. Specifically, the process involves summing up the total salt content using numerical calculations.

[0537] Step 6:

[0538] The server notifies the user via an information notification device based on the total salt content result. The input is the calculated total salt content, and the output is a notification message displayed on the user's terminal. The server generates the notification content and sends the message to the user.

[0539] Step 7:

[0540] The server determines whether the calculated total salt content exceeds the set standard, and if so, issues an alarm via a signaling device. The inputs are the total salt content and the set standard, and the output is a warning signal. The server compares the value to the standard and, if necessary, generates a warning signal and sends it to the terminal.

[0541] Step 8:

[0542] The server stores analysis and calculation results in an information storage device and records them as long-term food intake history data. The input is a series of analysis and calculation results, and the output is historical data stored in a database. The server stores the information through a data management system.

[0543] (Application Example 1)

[0544] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0545] Managing salt intake in meals smoothly for elderly people living alone is a challenge. Existing methods make it difficult to effectively utilize information even when photos of meals are taken, requiring continuous support from family members or caregivers. Furthermore, the inability to visually manage salt intake poses a risk of problems. Therefore, there is a need for a system that allows users to easily understand their salt intake and manage their health while sharing information with relatives.

[0546] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0547] In this invention, the server includes image analysis means for receiving and analyzing images of photographed meals, visualization means for displaying the user's long-term dietary salt intake in a chart format, and remote access means for enabling information sharing among multiple users. This makes it possible for users to easily manage their salt intake and manage their health while sharing information with relatives.

[0548] "Image analysis means" refers to a process used to receive images of a photographed meal and identify identifiable food items.

[0549] The "salt content calculation method" is a process that retrieves the salt content of food items identified based on the analyzed images from a database and then sums up the total salt content.

[0550] The "notification generation method" is a procedure for notifying the user's device of the calculated amount of salt.

[0551] A "warning signaling mechanism" is a system that issues a warning signal when the salt content exceeds the standard.

[0552] A "data storage method" is a process for storing analysis results and calculation results over a long period of time so that they can be analyzed later.

[0553] "Visualization means" refers to a function that displays the user's long-term salt intake data in a chart format.

[0554] "Remote access means" refers to technologies that enable multiple users to share and access information via the internet.

[0555] "Family notification method" refers to a method of notifying the user's relatives of the photographed meal images so that they can view them remotely.

[0556] "Data analysis tools" refer to tools used to analyze users' salt intake over time.

[0557] This invention provides a system that allows elderly people to easily manage their salt intake through meals. The system begins when a user takes a picture of their meal using a smartphone or other device. The device sends the captured image to a server, which then performs image analysis on the received image. The image analysis utilizes a deep learning model, enabling automatic identification of food items. For each identified food item, the server retrieves the salt content from a database and calculates the total salt intake.

[0558] The calculation results are notified to the user's device and to relatives located remotely. The notification includes information on the amount of salt in each food item and the total amount of salt. If the salt intake exceeds a pre-set standard, a signal is sent to the device to emit a warning sound. Furthermore, the server stores the analysis and calculation results as data for use in long-term health management. This stored data is visualized in various charts and graphs, making it easy to check salt intake over time. This also allows relatives to provide dietary advice remotely.

[0559] As a concrete example, by using a prompt such as, "Take a picture of your lunch, is today's salt intake okay?", users can easily enjoy the benefits of the invention. This system uses software such as Python and TensorFlow to process and calculate data. Smartphones and tablets are commonly used as hardware. In this way, the present invention provides comprehensive support for easily managing the amount of salt consumed from daily meals.

[0560] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0561] Step 1:

[0562] A user takes a picture of their meal using their smartphone camera. The captured image is saved on the device and then sent to a server via the internet. In this process, the image is the input, and the data transmission to the server is the output.

[0563] Step 2:

[0564] The server analyzes the received images using a deep learning model. Specifically, each food item in the image is identified. A list of labeled food items is output from the image data. An object recognition algorithm is used in this process.

[0565] Step 3:

[0566] The server retrieves the salt content of each food item from the database based on the label. It takes a list of foods as input and outputs the corresponding salt content information. Data retrieval is performed here using SQL queries, etc.

[0567] Step 4:

[0568] The salt content calculation method calculates the total salt content by summing up the acquired salt amounts. An array containing the individual salt amounts is taken as input, and the total salt content is output. The calculation is performed using standard addition.

[0569] Step 5:

[0570] The calculated total salt intake is sent as a notification to the user's device and their relatives' devices. The notification message is output using the total salt intake data as input. A messaging API is used for this process.

[0571] Step 6:

[0572] The server stores the calculation results in a database, accumulating them as a long-term dietary history. Here, the analysis results and calculation results serve as input, and the accumulated data records are output. A database management system is used for data storage.

[0573] Step 7:

[0574] The system visualizes accumulated data in graphs and charts, allowing users to review their past salt intake history. Historical data retrieved from the database is used as input, and visualized information is output. Data analysis software is used for visualization.

[0575] Step 8:

[0576] If the user-set threshold is exceeded, a signal is sent to the device to emit a warning sound. The total salt content is used as input to output the warning sound. This process utilizes the device's notification system.

[0577] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0578] This invention aims to more effectively manage the health of the elderly by combining a salt management system for meals with a user emotion recognition function. Specific embodiments are described below.

[0579] The system initiates the process by having the user take a picture of their meal with their smartphone or a dedicated device. The captured image is saved on the device and uploaded to the server. The server uses image analysis to identify the food in the received image and retrieves salt content information from a database for each recognized food item.

[0580] Subsequently, the server uses a salt content calculation device to sum up the identified salt amounts and calculate the total salt content. The calculation result is sent as a message to the user and their relatives by a notification generation device. This message includes the salt content of individual foods and an indicator tailored to the newly adopted emotion.

[0581] The emotion engine, a key feature of this invention, has the ability to recognize the user's emotions in real time. The user's emotions are captured through camera or voice input and analyzed by an emotion recognition algorithm. Based on the results of this emotion analysis, notifications and warnings regarding salt intake are incorporated. For example, if the user is feeling stressed, the notification is also sent to relatives, including advice that takes stress reduction into consideration.

[0582] Furthermore, emotional data is stored in a data storage system along with the user's dietary history. This data is used by data analysis tools to analyze the user's salt intake over time, supporting long-term health management. In addition, fluctuations based on the user's emotional data are recorded over a long period and provided to family members. This enables comprehensive health management, including the user's psychological state.

[0583] For example, if a user feels anxious during a meal, the system recognizes this through an emotion engine. The server analyzes this and sends advice to the family, along with a notification about the amount of salt consumed, suggesting ways to relax and alleviate the anxiety. Based on this, the family can provide appropriate support to the user.

[0584] Thus, the present invention provides a system that supports a healthier lifestyle for the elderly by simultaneously managing salt intake and the user's emotions.

[0585] The following describes the processing flow.

[0586] Step 1:

[0587] The user takes a photo of their meal using their smartphone or a dedicated device. This triggers the system to start operating.

[0588] Step 2:

[0589] The device temporarily stores the captured images and uploads them to the server via the network. The device is configured to automatically upload images after they are captured.

[0590] Step 3:

[0591] The server analyzes the received images based on image analysis tools to identify the food items in the photographs. AI-based image recognition technology is used for food identification.

[0592] Step 4:

[0593] The server retrieves the salt content of each food item from a database based on the data of the identified food item. This makes it possible to collect accurate salt information.

[0594] Step 5:

[0595] The server uses a salt content calculation device to add up the salt content of each food item and calculate the total salt content. This value will be used in subsequent notifications.

[0596] Step 6:

[0597] To recognize the user's emotions in real time, the device uses a camera or microphone to acquire emotional data from the user's facial expressions and voice.

[0598] Step 7:

[0599] Emotional data acquired from the terminal is sent to a server, where an emotion engine processes it and determines the user's emotions.

[0600] Step 8:

[0601] The server combines salt levels and emotional data to generate a notification message. This message includes advice based on the salt level and the corresponding emotional state.

[0602] Step 9:

[0603] The server sends the generated notifications to the user's and registered relatives' devices. It provides not only salt intake calculation results but also emotionally-driven advice.

[0604] Step 10:

[0605] If the server detects a salt level exceeding the standard or any resulting emotional changes, it will use a warning system on the terminal to emit a warning sound or message.

[0606] Step 11:

[0607] The server stores analysis results and emotional data in a data storage system. This data is used for long-term health management and tracking changes in the user's emotional state.

[0608] Step 12:

[0609] The stored data is analyzed using data analysis tools. This makes it easier to understand the user's health status, salt intake, and emotional state trends.

[0610] This system allows for the simultaneous management of the dietary habits and emotional well-being of elderly individuals, enabling more comprehensive health management.

[0611] (Example 2)

[0612] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0613] For elderly individuals, managing salt intake is crucial for maintaining health. However, accurately determining the salt content of meals is generally difficult. Furthermore, the impact of emotional state during meals on health should also be considered. Conventional systems have not comprehensively considered both salt management and the user's emotional state. Therefore, there is a need for technology that can simultaneously achieve appropriate management of dietary content and care for emotional state.

[0614] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0615] In this invention, the server includes an image processing means for receiving and analyzing images of photographed meals, a quantity calculation means for identifying food from the analyzed images and obtaining and summing the salt content from an information storage device, and an emotion analysis means for recognizing the user's emotions and reflecting them in the analysis results and notifications. This enables comprehensive health management that accurately manages the amount of salt in the user's meals while also taking into account their emotional state.

[0616] "Image processing means" refers to technology that provides functions for receiving images of photographed food and analyzing those images.

[0617] The "quantity calculation means" is a technology for obtaining and summing the salt content of food identified from the analyzed images by retrieving it from an information storage device.

[0618] "Notification generation means" refers to a technology equipped with a function for transmitting calculated salt content information to a notification device.

[0619] A "warning issuance method" is a technology that provides a method for issuing a warning when the salt content exceeds the standard.

[0620] "Information storage means" refers to technology for accumulating analysis results and calculation results to store dietary information over a long period of time.

[0621] "Emotion analysis means" refers to technology that recognizes the user's emotions and reflects them in the analysis results and notifications.

[0622] "Notification adjustment means" refers to technology that has a function to adjust notification content based on the user's emotional state.

[0623] "Family notification means" refers to a technology that notifies a remote relative of a captured image, allowing that relative to view it.

[0624] "Information analysis means" refers to technology for performing analysis based on the amount of salt consumed by the user and the time course of their emotional state.

[0625] This invention is a system that supports health management by comprehensively analyzing a user's dietary management and emotional state. Specific embodiments are described below.

[0626] Users take photos of their meals using their smartphones or dedicated terminals. These terminals store the images and upload them to a server via the network. The server analyzes the received images using image processing software (e.g., OpenCV or TensorFlow) to identify the food items. Machine learning algorithms are used in this process, based on the food's shape, color, patterns, and other characteristics.

[0627] The server calculates the salt content using information about the identified food items. This information is obtained from a nutritional database. The server then sends the total salt content to the user and distant relatives via a notification generation system. This notification includes details of the meal and the total salt content.

[0628] Furthermore, the user's emotions are captured using the device's camera and microphone. These are processed in real time by emotion analysis algorithms (e.g., Amazon Rekognition or Google Cloud Speech-to-Text). Based on the recognized emotions, the server adjusts the notification content and sends it to the relative, including appropriate advice.

[0629] This system stores users' dietary history and emotional data using information storage mechanisms for long-term use. The data is analyzed using information analysis mechanisms to track changes in the user's salt intake and emotional state over time. This enables comprehensive health management.

[0630] For example, if a user feels down while eating a fish dish at dinner, the system recognizes this situation and sends a notification to a relative that includes advice on how to improve their mood, along with information on the amount of salt consumed.

[0631] An example of a prompt to input into the generative AI model is as follows: "I would like to detail the specific processes of a health management system for the elderly. This includes everything from taking photos of meals to managing salt intake, and also mentions the role of sentiment analysis."

[0632] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0633] Step 1:

[0634] The user takes a picture of their meal using a smartphone or dedicated device. The captured image is saved to the device. The input is image data generated by the user, and the output is an image file stored on the device. Specifically, the camera application is used to capture the image and save it to the device's storage.

[0635] Step 2:

[0636] The terminal uploads the saved image to the server via the network. The input is the image file saved in step 1, and the output is the image data transferred to the server. Specifically, the image file is sent to a specified endpoint on the server via the internet connection.

[0637] Step 3:

[0638] The server analyzes the received images through image processing tools. It uses software such as OpenCV and TensorFlow to identify food items within the images. The input is image data uploaded to the server, and the output is identification information for the food items in the images. Specifically, it runs a machine learning model to analyze shape, color, and pattern to identify the food items.

[0639] Step 4:

[0640] The server retrieves the salt content of identified foods from a database and sums them up. The input is a list of identified foods, and the output is the salt content of each food and the sum of those salt content values. Specifically, it sends a query to the database for each food and adds up the retrieved salt content data.

[0641] Step 5:

[0642] The server generates a message containing the total salt content and food details, and sends it to the user and their relatives. The input is the calculated total salt content and food details, and the output is the notification message. Specifically, it uses a notification generation engine to create and send emails and app notifications.

[0643] Step 6:

[0644] The device recognizes the user's emotions in real time using its camera and microphone. Input is the user's facial expressions and voice data, while output is emotional state information. Specifically, it runs an emotion recognition algorithm to classify the emotional state.

[0645] Step 7:

[0646] The server adjusts notification content based on emotional state information. The input is the recognized emotional state, and the output is a notification message adjusted according to that emotional state. Specifically, if the user is stressed, the server generates a message suggesting ways to relax.

[0647] Step 8:

[0648] The server stores the user's meal history and emotional data through an information storage system. Inputs are the amount of salt consumed and emotional data for each meal, and output is the accumulated dataset. Specifically, it writes data to a database in preparation for long-term analysis.

[0649] Step 9:

[0650] The server analyzes the accumulated data using information analysis tools to extract temporal trends in the user's health status. The input is the accumulated dataset, and the output is the analysis results regarding the temporal changes in the user's salt intake and emotional state. Specifically, it runs a data analysis algorithm and generates a report.

[0651] (Application Example 2)

[0652] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0653] Excessive salt intake in the diet of the elderly is highly likely to have adverse effects on their health, and their mental state is also an important factor in health management. However, conventional dietary management systems have difficulty simultaneously managing salt intake and emotional state, and a more comprehensive health management system is needed. Furthermore, providing this information to family members living remotely is expected to provide better support.

[0654] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0655] In this invention, the server includes a digital image analysis means for receiving and analyzing images of photographed meals, a nutrition calculation means for identifying food from the analyzed images and retrieving and summing its salt content from a database, and an emotion recognition means for recognizing the user's emotional state and reflecting this in notifications and warnings regarding salt intake. This enables more comprehensive health management by simultaneously managing salt intake in meals and the emotional state of elderly people, and by providing information to family members in remote locations.

[0656] "Digital image analysis means" refers to technology that receives images of food, analyzes their contents, and identifies the food items.

[0657] The "nutrition calculation method" is a technique that retrieves the amount of salt in foods identified by image analysis from a database and sums it up.

[0658] "Information generation means" refers to a technology that generates and transmits information to a notification device based on the calculated amount of salt.

[0659] An "alert issuance system" is a technology that issues a warning to users and relevant parties when a salt level exceeding the standard is detected.

[0660] "Information storage means" refers to technology that stores analysis results and calculation results as data and manages data on eating habits over a long period of time.

[0661] "Emotion recognition means" refers to technology that recognizes the user's emotional state and reflects it in notifications and warnings regarding salt intake.

[0662] "Family notification means" refers to technology that notifies relatives in remote locations of meal and emotional data, making it possible for them to review and verify it.

[0663] "Data analysis means" refers to technology that enables analysis based on the user's salt intake and emotional state over time.

[0664] This invention is designed as a health management system that simultaneously considers the salt intake management and emotional state of elderly individuals. Users can use a smartphone or tablet device to take photos of each meal and send the images to the server in real time.

[0665] The server uses tools such as "OpenCV" and "TensorFlow" as image analysis tools to identify food items from received images of meals. Based on the food information obtained through this analysis, it retrieves the amount of salt for each food item from an "SQL database" as a nutrition calculation tool and sums them up. The resulting salt amount is then notified in real time via a notification generation tool to the user's smartphone application or the devices of registered relatives.

[0666] In addition, video and audio captured through the user's camera or microphone are analyzed using emotion recognition technologies such as "Google Cloud Vision" and "Microsoft Azure Cognitive Services" to determine the user's emotional state. Notifications incorporating this emotional data provide personalized advice, including suggestions for relaxation methods if the user is feeling stressed.

[0667] The data storage method involves accumulating user salt intake and emotional data over a long period using a NoSQL database, among other methods. This data is then used by data analysis tools to analyze the correlation between salt intake and emotions over time.

[0668] For example, if a user chooses a dish high in salt for lunch, an image of it is sent to the system, and the amount of salt is immediately calculated. If the system detects that the user is experiencing stress during the meal, a notification is sent through the application to the user and their family members, such as, "This dish is high in salt. How about adding some vegetables? We have prepared some calming music for you to help you relax."

[0669] This system enables comprehensive management of users' health status, contributing to an improved quality of life for the elderly. Furthermore, it accepts user input in the form of prompts to be input into the generating AI model, such as, "Please generate suggestions if the meal Grandma is about to eat is high in salt. Also, please consider relaxation measures if she appears anxious."

[0670] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0671] Step 1:

[0672] The user takes a picture of their meal with their smartphone camera. The captured image is saved on the device and sent to the server. The input for this step is the image of the meal, and the output is the image data sent to the server.

[0673] Step 2:

[0674] The server analyzes the received image data using "OpenCV" to identify the food items within the image. This process takes the image data as input to determine the type of food item, which is then output. The identified information is then sent to the next processing step.

[0675] Step 3:

[0676] The server retrieves salt content information for each food item identified from the SQL database. The input is the identified food item information, and the output is the salt content data corresponding to each food item. This allows for an understanding of the overall nutritional components of each food item.

[0677] Step 4:

[0678] The salt content of each identified food item is summed up to calculate the total salt content. The input for this step is the salt content data for each food item, and the output is the total salt content. The server performs this calculation, providing the necessary basic data for notifying the user.

[0679] Step 5:

[0680] The server uses "Google Cloud Vision" to analyze video and audio acquired from the camera or microphone to recognize the user's emotional state. This process uses the user's video and audio input as the data source and outputs emotional state information.

[0681] Step 6:

[0682] The server uses a notification generation mechanism to create and send notifications to the user and registered relatives based on total salt intake and emotional state. The input is salt intake and emotional state, and the output is a pre-configured notification message. This process delivers notifications, including advice tailored to the user's purpose.

[0683] Step 7:

[0684] The notified data is stored in a NoSQL database and accumulated as a history of salt intake and emotional state. Inputs are dietary and emotional data, and the accumulated information is output. This enables long-term health management for the user.

[0685] Step 8:

[0686] The server uses data analysis tools to analyze accumulated data and interpret changes in salt intake and emotions over time. The input is accumulated historical data, and the output is the interpreted analysis results. This will provide insights that can be useful for future health management.

[0687] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0688] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0689] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0690] [Fourth Embodiment]

[0691] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0692] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0693] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0694] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0695] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0696] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0697] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0698] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0699] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0700] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0701] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0702] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0703] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] The system according to the present invention helps elderly people living alone to easily manage their salt intake, and a specific embodiment thereof will be described below.

[0705] The system starts when the user takes a picture of their meal with a smartphone or dedicated device. The captured image is saved on the device and sent to a server via the network. Upon receiving the image, the server uses image analysis to identify the food items. A deep learning model is used here to label the different food items in the image.

[0706] For each identified food item, the server retrieves information on the corresponding salt content from its internal database and uses a salt content calculation device to sum these values ​​and calculate the total salt content. This calculation result is then used by a notification generation device to generate a notification message to be sent to the user and their relatives who are remotely connected. This message includes the salt content for each food item and the total salt content.

[0707] If the salt intake exceeds a pre-set standard, the server sends a signal to the terminal via a warning system, instructing the terminal to sound a warning. Furthermore, the analysis and calculation results are stored in the server's database as the user's meal history by a data storage system.

[0708] Furthermore, by analyzing this accumulated data using data analysis tools, it becomes possible to help users manage their health in the long term. This allows users to easily understand the temporal trends in their salt intake, and relatives can remotely provide dietary advice.

[0709] As described above, the present invention allows users to easily manage the amount of salt they consume from their daily meals, and furthermore, enables relatives to provide appropriate advice based on that information. This makes it possible to provide a system that supports the maintenance of the health of the elderly.

[0710] The following describes the processing flow.

[0711] Step 1:

[0712] Users take photos of their meals using their smartphones or dedicated devices. The captured images are automatically saved to the device.

[0713] Step 2:

[0714] The device uploads the saved photos to a designated cloud server. The photo data is transferred using an internet connection during the upload process.

[0715] Step 3:

[0716] The server activates an image analysis system to analyze the received photo data. This process uses deep learning technology to identify each food item in the image.

[0717] Step 4:

[0718] The server retrieves salt content information for each identified food item from its internal database. It aggregates the salt content data for each food item and calculates the total salt content using a salt content calculation device.

[0719] Step 5:

[0720] The server uses the calculated total salt content and the salt content information of individual foods to generate a notification message. This notification message is created for the user and their relatives.

[0721] Step 6:

[0722] The server sends the generated notification message to the user's and their relatives' devices via a notification generation mechanism. Registered contact information is used for sending the message.

[0723] Step 7:

[0724] If the salt content exceeds a set standard, the server sends an instruction to the terminal using a warning system, and the terminal notifies the user by sounding a warning.

[0725] Step 8:

[0726] The server will save the analysis and calculation results in a data storage system. This data will be stored as the user's long-term dietary history.

[0727] Step 9:

[0728] The server uses the accumulated data to activate data analysis tools and perform an analysis of salt intake over time. The results of this analysis are provided periodically to the user and their relatives.

[0729] (Example 1)

[0730] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0731] Diet has a significant impact on health, and excessive salt intake, in particular, can increase health risks for the elderly. However, for the elderly, who increasingly eat alone, managing their daily salt intake appropriately is a difficult problem. Furthermore, even if relatives living far away want to support their daily diet, physical distance is often an obstacle. To address this, there is a need for a system that allows the elderly to easily and effectively manage their diet, and that enables relatives living far away to provide support.

[0732] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0733] In this invention, the server includes an analysis device means that receives a meal image captured from a location information terminal and detects food components by digital image processing; a nutritional analysis device means that extracts the amount of each component of the meal item from a data storage device based on the detected meal item and calculates its sum; and an information notification device means that transmits the calculated sum of component amounts to the user via a communication device. This allows the user to easily manage their daily salt intake, and enables relatives to remotely provide effective support for their diet.

[0734] A "location information terminal" is an electronic device used to capture images of food or other content and transmit them to a server.

[0735] "Digital image processing" refers to a technical method that analyzes received image data to detect food components.

[0736] "Food components" refer to the individual nutritional elements and their attributes contained in a meal.

[0737] An "analysis device" is a general term for hardware or software that analyzes the components and characteristics of food based on received data.

[0738] A "nutrition analyzer" is a device that calculates the amount of each component as a numerical value based on the detected food components and then calculates the sum of those values.

[0739] A "data storage device" refers to a database that stores information related to food ingredients and allows for searching and retrieving this information as needed.

[0740] An "information notification device" is a means of presenting information on the calculated sum of component amounts to the user, and functions as a communication device.

[0741] A "signal issuing device" is a device that generates alarms based on set criteria values ​​to alert the user.

[0742] An "information storage device" is a device that stores the results of analysis and calculations over a long period of time and manages historical data on food intake.

[0743] A "remote communication device" is a device or system that can transmit captured image information to an individual or group located at a distance.

[0744] A "data processing device" refers to a device that analyzes accumulated information and generates detailed reports based on changes over time.

[0745] This invention provides a system that allows elderly people living alone to easily manage their daily salt intake. The system starts when the user uses a location-based terminal to take a picture of their meal, saves the image to the terminal, and transmits it to a server via a network. The server analyzes the received image using digital image processing to detect food components. Various deep learning models specialized in image analysis are used for this purpose.

[0746] The server extracts component amounts from data storage based on the components obtained by the analysis device. These component amounts are then summed up using a nutritional analyzer to calculate the total salt intake. The calculated results are reported to the user via an information notification device. The notification includes the component amounts for each food item and the total component amounts, allowing the user to accurately control their intake.

[0747] If a user's salt intake exceeds a set limit, the server issues an alarm via a signaling device. Upon receiving the alarm, the terminal emits an audible or visual warning to alert the user. Analysis and calculation results are stored in an information storage device, and based on this historical data, the server uses a data processing device to analyze long-term nutritional intake trends and provide advice for maintaining health.

[0748] Furthermore, the captured image information can be shared with relatives in distant locations via a remote communication device, allowing them to provide direct support remotely. This sharing of information enables relatives to understand the user's daily eating habits and provide appropriate feedback.

[0749] A concrete example is a scenario where a user eats ramen and salad for lunch and takes a picture of it with their device. The server analyzes this image, identifies the salt content of each food item, and notifies the user of the results. The prompt used for the generating AI model is, "Take pictures of your ramen and salad for lunch and calculate the salt content of each." Through this prompt, the server can perform appropriate image analysis and provide information.

[0750] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0751] Step 1:

[0752] The user takes a picture of their meal using a smartphone or a dedicated camera device. The captured image is saved to the device. This image data becomes the input for the next process. Specifically, the user launches the camera app on their device, frames the meal, and presses the shutter button.

[0753] Step 2:

[0754] The terminal sends the saved image to the server over the network. Here, the input is the image file stored on the terminal, and the output is the image data received by the server. The terminal uses its communication function to upload the image file to the server according to the appropriate communication protocol.

[0755] Step 3:

[0756] The server analyzes the received image data using digital image processing technology. In this step, the image data transferred to the server is used as input, and label data identifying the food is generated as output. The server uses a generative AI model to extract the features of the food and label it.

[0757] Step 4:

[0758] The server searches the data storage device based on the analyzed label data to obtain the salt content of each food item. The input is food label data, and the output is the corresponding salt content information. The server queries the database using the name of the identified food item as a key to retrieve the corresponding salt content.

[0759] Step 5:

[0760] The server calculates the total salt content by summing up the salt content obtained using a nutritional analyzer. In this step, the salt content information of each food item is used as input, and the output is the calculated total salt content. Specifically, the process involves summing up the total salt content using numerical calculations.

[0761] Step 6:

[0762] The server notifies the user via an information notification device based on the total salt content result. The input is the calculated total salt content, and the output is a notification message displayed on the user's terminal. The server generates the notification content and sends the message to the user.

[0763] Step 7:

[0764] The server determines whether the calculated total salt content exceeds the set standard, and if so, issues an alarm via a signaling device. The inputs are the total salt content and the set standard, and the output is a warning signal. The server compares the value to the standard and, if necessary, generates a warning signal and sends it to the terminal.

[0765] Step 8:

[0766] The server stores analysis and calculation results in an information storage device and records them as long-term food intake history data. The input is a series of analysis and calculation results, and the output is historical data stored in a database. The server stores the information through a data management system.

[0767] (Application Example 1)

[0768] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0769] Managing salt intake in meals smoothly for elderly people living alone is a challenge. Existing methods make it difficult to effectively utilize information even when photos of meals are taken, requiring continuous support from family members or caregivers. Furthermore, the inability to visually manage salt intake poses a risk of problems. Therefore, there is a need for a system that allows users to easily understand their salt intake and manage their health while sharing information with relatives.

[0770] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0771] In this invention, the server includes image analysis means for receiving and analyzing images of photographed meals, visualization means for displaying the user's long-term dietary salt intake in a chart format, and remote access means for enabling information sharing among multiple users. This makes it possible for users to easily manage their salt intake and manage their health while sharing information with relatives.

[0772] "Image analysis means" refers to a process used to receive images of a photographed meal and identify identifiable food items.

[0773] The "salt content calculation method" is a process that retrieves the salt content of food items identified based on the analyzed images from a database and then sums up the total salt content.

[0774] The "notification generation method" is a procedure for notifying the user's device of the calculated amount of salt.

[0775] A "warning signaling mechanism" is a system that issues a warning signal when the salt content exceeds the standard.

[0776] A "data storage method" is a process for storing analysis results and calculation results over a long period of time so that they can be analyzed later.

[0777] "Visualization means" refers to a function that displays the user's long-term salt intake data in a chart format.

[0778] "Remote access means" refers to technologies that enable multiple users to share and access information via the internet.

[0779] "Family notification method" refers to a method of notifying the user's relatives of the photographed meal images so that they can view them remotely.

[0780] "Data analysis tools" refer to tools used to analyze users' salt intake over time.

[0781] This invention provides a system that allows elderly people to easily manage their salt intake through meals. The system begins when a user takes a picture of their meal using a smartphone or other device. The device sends the captured image to a server, which then performs image analysis on the received image. The image analysis utilizes a deep learning model, enabling automatic identification of food items. For each identified food item, the server retrieves the salt content from a database and calculates the total salt intake.

[0782] The calculation results are notified to the user's device and to relatives located remotely. The notification includes information on the amount of salt in each food item and the total amount of salt. If the salt intake exceeds a pre-set standard, a signal is sent to the device to emit a warning sound. Furthermore, the server stores the analysis and calculation results as data for use in long-term health management. This stored data is visualized in various charts and graphs, making it easy to check salt intake over time. This also allows relatives to provide dietary advice remotely.

[0783] As a concrete example, by using a prompt such as, "Take a picture of your lunch, is today's salt intake okay?", users can easily enjoy the benefits of the invention. This system uses software such as Python and TensorFlow to process and calculate data. Smartphones and tablets are commonly used as hardware. In this way, the present invention provides comprehensive support for easily managing the amount of salt consumed from daily meals.

[0784] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0785] Step 1:

[0786] A user takes a picture of their meal using their smartphone camera. The captured image is saved on the device and then sent to a server via the internet. In this process, the image is the input, and the data transmission to the server is the output.

[0787] Step 2:

[0788] The server analyzes the received images using a deep learning model. Specifically, each food item in the image is identified. A list of labeled food items is output from the image data. An object recognition algorithm is used in this process.

[0789] Step 3:

[0790] The server retrieves the salt content of each food item from the database based on the label. It takes a list of foods as input and outputs the corresponding salt content information. Data retrieval is performed here using SQL queries, etc.

[0791] Step 4:

[0792] The salt content calculation method calculates the total salt content by summing up the acquired salt amounts. An array containing the individual salt amounts is taken as input, and the total salt content is output. The calculation is performed using standard addition.

[0793] Step 5:

[0794] The calculated total salt intake is sent as a notification to the user's device and their relatives' devices. The notification message is output using the total salt intake data as input. A messaging API is used for this process.

[0795] Step 6:

[0796] The server stores the calculation results in a database, accumulating them as a long-term dietary history. Here, the analysis results and calculation results serve as input, and the accumulated data records are output. A database management system is used for data storage.

[0797] Step 7:

[0798] The system visualizes accumulated data in graphs and charts, allowing users to review their past salt intake history. Historical data retrieved from the database is used as input, and visualized information is output. Data analysis software is used for visualization.

[0799] Step 8:

[0800] If the user-set threshold is exceeded, a signal is sent to the device to emit a warning sound. The total salt content is used as input to output the warning sound. This process utilizes the device's notification system.

[0801] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0802] This invention aims to more effectively manage the health of the elderly by combining a salt management system for meals with a user emotion recognition function. Specific embodiments are described below.

[0803] The system initiates the process by having the user take a picture of their meal with their smartphone or a dedicated device. The captured image is saved on the device and uploaded to the server. The server uses image analysis to identify the food in the received image and retrieves salt content information from a database for each recognized food item.

[0804] Subsequently, the server uses a salt content calculation device to sum up the identified salt amounts and calculate the total salt content. The calculation result is sent as a message to the user and their relatives by a notification generation device. This message includes the salt content of individual foods and an indicator tailored to the newly adopted emotion.

[0805] The emotion engine, a key feature of this invention, has the ability to recognize the user's emotions in real time. The user's emotions are captured through camera or voice input and analyzed by an emotion recognition algorithm. Based on the results of this emotion analysis, notifications and warnings regarding salt intake are incorporated. For example, if the user is feeling stressed, the notification is also sent to relatives, including advice that takes stress reduction into consideration.

[0806] Furthermore, emotional data is stored in a data storage system along with the user's dietary history. This data is used by data analysis tools to analyze the user's salt intake over time, supporting long-term health management. In addition, fluctuations based on the user's emotional data are recorded over a long period and provided to family members. This enables comprehensive health management, including the user's psychological state.

[0807] For example, if a user feels anxious during a meal, the system recognizes this through an emotion engine. The server analyzes this and sends advice to the family, along with a notification about the amount of salt consumed, suggesting ways to relax and alleviate the anxiety. Based on this, the family can provide appropriate support to the user.

[0808] Thus, the present invention provides a system that supports a healthier lifestyle for the elderly by simultaneously managing salt intake and the user's emotions.

[0809] The following describes the processing flow.

[0810] Step 1:

[0811] The user takes a photo of their meal using their smartphone or a dedicated device. This triggers the system to start operating.

[0812] Step 2:

[0813] The device temporarily stores the captured images and uploads them to the server via the network. The device is configured to automatically upload images after they are captured.

[0814] Step 3:

[0815] The server analyzes the received images based on image analysis tools to identify the food items in the photographs. AI-based image recognition technology is used for food identification.

[0816] Step 4:

[0817] The server retrieves the salt content of each food item from a database based on the data of the identified food item. This makes it possible to collect accurate salt information.

[0818] Step 5:

[0819] The server uses a salt content calculation device to add up the salt content of each food item and calculate the total salt content. This value will be used in subsequent notifications.

[0820] Step 6:

[0821] To recognize the user's emotions in real time, the device uses a camera or microphone to acquire emotional data from the user's facial expressions and voice.

[0822] Step 7:

[0823] Emotional data acquired from the terminal is sent to a server, where an emotion engine processes it and determines the user's emotions.

[0824] Step 8:

[0825] The server combines salt levels and emotional data to generate a notification message. This message includes advice based on the salt level and the corresponding emotional state.

[0826] Step 9:

[0827] The server sends the generated notifications to the user's and registered relatives' devices. It provides not only salt intake calculation results but also emotionally-driven advice.

[0828] Step 10:

[0829] If the server detects a salt level exceeding the standard or any resulting emotional changes, it will use a warning system on the terminal to emit a warning sound or message.

[0830] Step 11:

[0831] The server stores analysis results and emotional data in a data storage system. This data is used for long-term health management and tracking changes in the user's emotional state.

[0832] Step 12:

[0833] The stored data is analyzed using data analysis tools. This makes it easier to understand the user's health status, salt intake, and emotional state trends.

[0834] This system allows for the simultaneous management of the dietary habits and emotional well-being of elderly individuals, enabling more comprehensive health management.

[0835] (Example 2)

[0836] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0837] For elderly individuals, managing salt intake is crucial for maintaining health. However, accurately determining the salt content of meals is generally difficult. Furthermore, the impact of emotional state during meals on health should also be considered. Conventional systems have not comprehensively considered both salt management and the user's emotional state. Therefore, there is a need for technology that can simultaneously achieve appropriate management of dietary content and care for emotional state.

[0838] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0839] In this invention, the server includes an image processing means for receiving and analyzing images of photographed meals, a quantity calculation means for identifying food from the analyzed images and obtaining and summing the salt content from an information storage device, and an emotion analysis means for recognizing the user's emotions and reflecting them in the analysis results and notifications. This enables comprehensive health management that accurately manages the amount of salt in the user's meals while also taking into account their emotional state.

[0840] "Image processing means" refers to technology that provides functions for receiving images of photographed food and analyzing those images.

[0841] The "quantity calculation means" is a technology for obtaining and summing the salt content of food identified from the analyzed images by retrieving it from an information storage device.

[0842] "Notification generation means" refers to a technology equipped with a function for transmitting calculated salt content information to a notification device.

[0843] A "warning issuance method" is a technology that provides a method for issuing a warning when the salt content exceeds the standard.

[0844] "Information storage means" refers to technology for accumulating analysis results and calculation results to store dietary information over a long period of time.

[0845] "Emotion analysis means" refers to technology that recognizes the user's emotions and reflects them in the analysis results and notifications.

[0846] "Notification adjustment means" refers to technology that has a function to adjust notification content based on the user's emotional state.

[0847] "Family notification means" refers to a technology that notifies a remote relative of a captured image, allowing that relative to view it.

[0848] "Information analysis means" refers to technology for performing analysis based on the amount of salt consumed by the user and the time course of their emotional state.

[0849] This invention is a system that supports health management by comprehensively analyzing a user's dietary management and emotional state. Specific embodiments are described below.

[0850] Users take photos of their meals using their smartphones or dedicated terminals. These terminals store the images and upload them to a server via the network. The server analyzes the received images using image processing software (e.g., OpenCV or TensorFlow) to identify the food items. Machine learning algorithms are used in this process, based on the food's shape, color, patterns, and other characteristics.

[0851] The server calculates the salt content using information about the identified food items. This information is obtained from a nutritional database. The server then sends the total salt content to the user and distant relatives via a notification generation system. This notification includes details of the meal and the total salt content.

[0852] Furthermore, the user's emotions are captured using the device's camera and microphone. These are processed in real time by emotion analysis algorithms (e.g., Amazon Rekognition or Google Cloud Speech-to-Text). Based on the recognized emotions, the server adjusts the notification content and sends it to the relative, including appropriate advice.

[0853] This system stores users' dietary history and emotional data using information storage mechanisms for long-term use. The data is analyzed using information analysis mechanisms to track changes in the user's salt intake and emotional state over time. This enables comprehensive health management.

[0854] For example, if a user feels down while eating a fish dish at dinner, the system recognizes this situation and sends a notification to a relative that includes advice on how to improve their mood, along with information on the amount of salt consumed.

[0855] An example of a prompt to input into the generative AI model is as follows: "I would like to detail the specific processes of a health management system for the elderly. This includes everything from taking photos of meals to managing salt intake, and also mentions the role of sentiment analysis."

[0856] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0857] Step 1:

[0858] The user takes a picture of their meal using a smartphone or dedicated device. The captured image is saved to the device. The input is image data generated by the user, and the output is an image file stored on the device. Specifically, the camera application is used to capture the image and save it to the device's storage.

[0859] Step 2:

[0860] The terminal uploads the saved image to the server via the network. The input is the image file saved in step 1, and the output is the image data transferred to the server. Specifically, the image file is sent to a specified endpoint on the server via the internet connection.

[0861] Step 3:

[0862] The server analyzes the received images through image processing tools. It uses software such as OpenCV and TensorFlow to identify food items within the images. The input is image data uploaded to the server, and the output is identification information for the food items in the images. Specifically, it runs a machine learning model to analyze shape, color, and pattern to identify the food items.

[0863] Step 4:

[0864] The server retrieves the salt content of identified foods from a database and sums them up. The input is a list of identified foods, and the output is the salt content of each food and the sum of those salt content values. Specifically, it sends a query to the database for each food and adds up the retrieved salt content data.

[0865] Step 5:

[0866] The server generates a message containing the total salt content and food details, and sends it to the user and their relatives. The input is the calculated total salt content and food details, and the output is the notification message. Specifically, it uses a notification generation engine to create and send emails and app notifications.

[0867] Step 6:

[0868] The device recognizes the user's emotions in real time using its camera and microphone. Input is the user's facial expressions and voice data, while output is emotional state information. Specifically, it runs an emotion recognition algorithm to classify the emotional state.

[0869] Step 7:

[0870] The server adjusts notification content based on emotional state information. The input is the recognized emotional state, and the output is a notification message adjusted according to that emotional state. Specifically, if the user is stressed, the server generates a message suggesting ways to relax.

[0871] Step 8:

[0872] The server stores the user's meal history and emotional data through an information storage system. Inputs are the amount of salt consumed and emotional data for each meal, and output is the accumulated dataset. Specifically, it writes data to a database in preparation for long-term analysis.

[0873] Step 9:

[0874] The server analyzes the accumulated data using information analysis tools to extract temporal trends in the user's health status. The input is the accumulated dataset, and the output is the analysis results regarding the temporal changes in the user's salt intake and emotional state. Specifically, it runs a data analysis algorithm and generates a report.

[0875] (Application Example 2)

[0876] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0877] Excessive salt intake in the diet of the elderly is highly likely to have adverse effects on their health, and their mental state is also an important factor in health management. However, conventional dietary management systems have difficulty simultaneously managing salt intake and emotional state, and a more comprehensive health management system is needed. Furthermore, providing this information to family members living remotely is expected to provide better support.

[0878] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0879] In this invention, the server includes a digital image analysis means for receiving and analyzing images of photographed meals, a nutrition calculation means for identifying food from the analyzed images and retrieving and summing its salt content from a database, and an emotion recognition means for recognizing the user's emotional state and reflecting this in notifications and warnings regarding salt intake. This enables more comprehensive health management by simultaneously managing salt intake in meals and the emotional state of elderly people, and by providing information to family members in remote locations.

[0880] "Digital image analysis means" refers to technology that receives images of food, analyzes their contents, and identifies the food items.

[0881] The "nutrition calculation method" is a technique that retrieves the amount of salt in foods identified by image analysis from a database and sums it up.

[0882] "Information generation means" refers to a technology that generates and transmits information to a notification device based on the calculated amount of salt.

[0883] An "alert issuance system" is a technology that issues a warning to users and relevant parties when a salt level exceeding the standard is detected.

[0884] "Information storage means" refers to technology that stores analysis results and calculation results as data and manages data on eating habits over a long period of time.

[0885] "Emotion recognition means" refers to technology that recognizes the user's emotional state and reflects it in notifications and warnings regarding salt intake.

[0886] "Family notification means" refers to technology that notifies relatives in remote locations of meal and emotional data, making it possible for them to review and verify it.

[0887] "Data analysis means" refers to technology that enables analysis based on the user's salt intake and emotional state over time.

[0888] This invention is designed as a health management system that simultaneously considers the salt intake management and emotional state of elderly individuals. Users can use a smartphone or tablet device to take photos of each meal and send the images to the server in real time.

[0889] The server uses tools such as "OpenCV" and "TensorFlow" as image analysis tools to identify food items from received images of meals. Based on the food information obtained through this analysis, it retrieves the amount of salt for each food item from an "SQL database" as a nutrition calculation tool and sums them up. The resulting salt amount is then notified in real time via a notification generation tool to the user's smartphone application or the devices of registered relatives.

[0890] In addition, video and audio captured through the user's camera or microphone are analyzed using emotion recognition technologies such as "Google Cloud Vision" and "Microsoft Azure Cognitive Services" to determine the user's emotional state. Notifications incorporating this emotional data provide personalized advice, including suggestions for relaxation methods if the user is feeling stressed.

[0891] The data storage method involves accumulating user salt intake and emotional data over a long period using a NoSQL database, among other methods. This data is then used by data analysis tools to analyze the correlation between salt intake and emotions over time.

[0892] For example, if a user chooses a dish high in salt for lunch, an image of it is sent to the system, and the amount of salt is immediately calculated. If the system detects that the user is experiencing stress during the meal, a notification is sent through the application to the user and their family members, such as, "This dish is high in salt. How about adding some vegetables? We have prepared some calming music for you to help you relax."

[0893] This system enables comprehensive management of users' health status, contributing to an improved quality of life for the elderly. Furthermore, it accepts user input in the form of prompts to be input into the generating AI model, such as, "Please generate suggestions if the meal Grandma is about to eat is high in salt. Also, please consider relaxation measures if she appears anxious."

[0894] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0895] Step 1:

[0896] The user takes a picture of their meal with their smartphone camera. The captured image is saved on the device and sent to the server. The input for this step is the image of the meal, and the output is the image data sent to the server.

[0897] Step 2:

[0898] The server analyzes the received image data using "OpenCV" to identify the food items within the image. This process takes the image data as input to determine the type of food item, which is then output. The identified information is then sent to the next processing step.

[0899] Step 3:

[0900] The server retrieves salt content information for each food item identified from the SQL database. The input is the identified food item information, and the output is the salt content data corresponding to each food item. This allows for an understanding of the overall nutritional components of each food item.

[0901] Step 4:

[0902] The salt content of each identified food item is summed up to calculate the total salt content. The input for this step is the salt content data for each food item, and the output is the total salt content. The server performs this calculation, providing the necessary basic data for notifying the user.

[0903] Step 5:

[0904] The server uses "Google Cloud Vision" to analyze video and audio acquired from the camera or microphone to recognize the user's emotional state. This process uses the user's video and audio input as the data source and outputs emotional state information.

[0905] Step 6:

[0906] The server uses a notification generation mechanism to create and send notifications to the user and registered relatives based on total salt intake and emotional state. The input is salt intake and emotional state, and the output is a pre-configured notification message. This process delivers notifications, including advice tailored to the user's purpose.

[0907] Step 7:

[0908] The notified data is stored in a NoSQL database and accumulated as a history of salt intake and emotional state. Inputs are dietary and emotional data, and the accumulated information is output. This enables long-term health management for the user.

[0909] Step 8:

[0910] The server uses data analysis tools to analyze accumulated data and interpret changes in salt intake and emotions over time. The input is accumulated historical data, and the output is the interpreted analysis results. This will provide insights that can be useful for future health management.

[0911] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0912] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0913] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0914] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0915] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0916] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0917] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0918] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0919] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0920] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0921] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0922] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0923] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0924] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0925] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0926] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0927] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0928] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0929] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0930] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0931] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0932] The following is further disclosed regarding the embodiments described above.

[0933] (Claim 1)

[0934] An image analysis means that receives and analyzes images of a photographed meal,

[0935] A salt content calculation means that identifies food from the analyzed image, retrieves its salt content from a database, and sums it up,

[0936] A notification generation means that sends the calculated salt amount to a notification device,

[0937] A warning system that issues a warning when the salt content exceeds the standard,

[0938] Data storage means for accumulating analysis results and calculation results and storing long-term dietary data.

[0939] A system that includes this.

[0940] (Claim 2)

[0941] The system according to claim 1, comprising a relative notification means for notifying a remote relative of the captured image and enabling them to review it.

[0942] (Claim 3)

[0943] The system according to claim 1, comprising data analysis means that enables analysis based on the amount of salt ingested by a user over time.

[0944] "Example 1"

[0945] (Claim 1)

[0946] An analysis device means that receives a food image captured from a location information terminal and detects food components by digital image processing,

[0947] A nutritional analysis device means that extracts the amount of each component of the detected dietary item from a data storage device and calculates its sum,

[0948] Information notification device means that transmits the calculated total component amount to the user via a communication device,

[0949] A signal issuing device means that activates an alarm when a component amount exceeding a set standard value is detected,

[0950] An information storage device that stores analysis results and calculated information in chronological order, recording a long-term history of food intake.

[0951] A system that includes this.

[0952] (Claim 2)

[0953] The system according to claim 1, comprising a remote communication device that transmits captured image information to relatives located in a remote location for viewing.

[0954] (Claim 3)

[0955] The system according to claim 1, comprising a data processing device that provides a detailed analysis based on the temporal changes in a user's nutrient intake.

[0956] "Application Example 1"

[0957] (Claim 1)

[0958] An image analysis means that receives and analyzes images of a photographed meal,

[0959] A salt content calculation means that identifies food from the analyzed image, retrieves its salt content from a database, and sums it up,

[0960] A notification generation means that sends the calculated salt amount to a notification device,

[0961] A warning system that issues a warning when the salt content exceeds the standard,

[0962] A data storage means for accumulating analysis results and calculation results, and for storing long-term dietary data,

[0963] A visualization method that displays a user's long-term dietary salt intake in a chart format,

[0964] A remote access method that enables information sharing by multiple users,

[0965] A system that includes this.

[0966] (Claim 2)

[0967] The system according to claim 1, comprising a relative notification means for notifying a remote relative of the captured image and enabling them to review it.

[0968] (Claim 3)

[0969] The system according to claim 1, comprising data analysis means that enables analysis based on the amount of salt ingested by a user over time.

[0970] "Example 2 of combining an emotion engine"

[0971] (Claim 1)

[0972] Image processing means for receiving and analyzing images of photographed meals,

[0973] A quantity calculation means that identifies food from analyzed images, obtains the amount of salt from an information storage device and sums it up,

[0974] A notification generation means that transmits the calculated salt amount to a notification device,

[0975] A warning system that issues a warning when the salt content exceeds the standard,

[0976] An information storage means for accumulating analysis results and calculation results and storing long-term dietary information,

[0977] A sentiment analysis method that recognizes the user's emotions and reflects them in the analysis results and notifications,

[0978] Notification adjustment means that adjusts notification content based on the user's emotional state.

[0979] A system that includes this.

[0980] (Claim 2)

[0981] The system according to claim 1, further comprising a relative notification means for notifying and allowing remote relatives to review the captured images.

[0982] (Claim 3)

[0983] The system according to claim 1, comprising information analysis means that enables analysis based on the amount of salt ingested by the user and the time course of their emotional state.

[0984] "Application example 2 when combining with an emotional engine"

[0985] (Claim 1)

[0986] A digital image analysis means that receives and analyzes images of photographed meals,

[0987] A nutritional calculation method that identifies food from analyzed images and retrieves and sums its salt content from a database,

[0988] Information generation means for transmitting the calculated amount of salt to a notification device,

[0989] An alert system that issues warnings when the amount of salt exceeds the standard,

[0990] An information storage means for accumulating analysis results and calculation results, and for storing long-term dietary habit data,

[0991] An emotion recognition system that recognizes the user's emotional state and reflects it in notifications and warnings regarding salt intake.

[0992] A system that includes this.

[0993] (Claim 2)

[0994] The system according to claim 1, comprising a family notification means for notifying and allowing relatives in a remote location to confirm captured images and emotional states.

[0995] (Claim 3)

[0996] The system according to claim 1, comprising data analysis means that enables analysis based on the amount of salt ingested by the user and the time course of their emotional state. [Explanation of Symbols]

[0997] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An image analysis means that receives and analyzes images of a photographed meal, A salt content calculation means that identifies food from the analyzed image, retrieves its salt content from a database, and sums it up, A notification generation means that sends the calculated salt amount to a notification device, A warning system that issues a warning when the salt content exceeds the standard, A data storage means for accumulating analysis results and calculation results, and for storing long-term dietary data, A visualization method that displays a user's long-term dietary salt intake in a chart format, A remote access method that enables information sharing by multiple users, A system that includes this.

2. The system according to claim 1, further comprising a relative notification means for notifying a remote relative of the captured image and enabling them to confirm it.

3. The system according to claim 1, comprising data analysis means that enables analysis based on the amount of salt ingested by a user over time.

Images