System

Abstract

A system is provided.SOLUTION: A system comprising means for photographing a meal of a person to be cared for, means for transmitting photographed image data to a server, means for analyzing the image data and recognizing types and quantities of food materials, means for recording a recognition result in a database, means for predicting necessary food materials and quantities thereof based on the recorded meal data, means for generating a next food material order list based on the prediction, means for automatically placing an order with a food material supplier based on the order list, and means for notifying a user of order information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People who require care often have difficulty shopping for the ingredients they need, and if they have dementia, it can be difficult to determine what they need. This often requires the assistance of family members or caregivers with shopping, but this is often difficult. Maintaining an appropriate nutritional balance also requires specialized knowledge, which places a significant burden on family members and caregivers. This invention aims to reduce the burden on family members and caregivers by efficiently managing the daily diet of people who require care, automatically securing the necessary ingredients, and maintaining a balanced diet. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for photographing the meals of a person requiring care, a means for transmitting the photographed image data to a server, a means for analyzing the image data to recognize the types and amounts of ingredients, a means for recording the recognition results in a database, a means for predicting the ingredients needed and their amounts based on the recorded meal data, a means for generating a next ingredient ordering list based on the prediction, a means for automatically placing orders with ingredient suppliers based on the ordering list, and a means for notifying the user of the ordering information.

[0006] In addition, to manage nutritional balance, we provide a system that analyzes the nutritional intake status of the person requiring care, evaluates the nutritional balance, and adds a means to generate improvement suggestions if the nutritional balance is inappropriate, and includes a means to notify the user of the suggestions. Furthermore, by analyzing past consumption patterns based on food records and providing a means to predict future consumption, we achieve more accurate food ingredient management. This allows for effective management of the dietary habits of the person requiring care and reduces the burden on family members and caregivers.

[0007] A "person requiring care" is someone who requires assistance from others in their daily lives.

[0008] The "means for photographing meals" refers to a device or method for acquiring image data of meals of a person requiring care using a camera device, a robot, or the like.

[0009] The "means for transmitting image data" refers to a function or device that transmits acquired image data to a server via a network.

[0010] "Means for analyzing image data to recognize the types and amounts of ingredients" refers to algorithms or software that use image analysis technology to identify the types and amounts of ingredients from food images.

[0011] A "database" is a storage system for registering and managing dietary records and analysis results.

[0012] "Recorded dietary data" refers to information about the dietary contents of a care recipient that is stored in a database.

[0013] "Means for predicting the ingredients needed and their quantities" refers to algorithms or software that calculate the ingredients needed in the future and their quantities based on past dietary data.

[0014] The "means for generating the next ingredient order list" refers to a function or device that automatically creates an order list based on the predicted types and required quantities of ingredients.

[0015] "Means for automatically placing orders with food ingredient suppliers" refers to a function or system that automatically sends food ingredient ordering requests to affiliated suppliers based on the generated ordering list.

[0016] "Means for notifying the user of order information" refers to a communication means or notification system for informing family members or caregivers of the order contents.

[0017] "Means for analyzing nutritional intake status" refers to algorithms or software that evaluate the nutritional intake status of individuals requiring care based on recorded dietary data.

[0018] "Means for evaluating nutritional balance and generating improvement suggestions if inappropriate" refers to a function or system that evaluates nutritional balance based on the analysis results and automatically generates improvement suggestions for nutrients that are deficient or in excess.

[0019] "Means for notifying users of suggestions" refers to communication means or notification systems for informing family members or caregivers of the generated improvement suggestions.

[0020] "Means for analyzing consumption patterns" refers to algorithms or software that analyze past dietary data and identify food consumption patterns.

[0021] "Means for predicting future consumption" refers to algorithms or software that predict future food consumption based on past consumption patterns. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention is a system that recognizes the daily dietary habits of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are as follows:

[0044] 1. Record and recognize food

[0045] Terminal

[0046] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[0047] The terminal transmits the stored image data to the server.

[0048] server

[0049] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[0050] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0051] Specific examples

[0052] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[0053] 2. Predicting necessary ingredients

[0054] server

[0055] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[0056] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[0057] Specific examples

[0058] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[0059] 3.Automatic ordering of ingredients

[0060] server

[0061] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[0062] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[0063] User

[0064] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0065] Specific examples

[0066] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[0067] 4. Nutritional balance management

[0068] server

[0069] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[0070] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[0071] The generated nutritional balance improvement proposal is notified to the user.

[0072] User

[0073] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0074] Specific examples

[0075] If the past week's records show that the care recipient has an iron deficiency, the server will send advice to the family member's smartphone, such as "We recommend adding iron-rich foods such as liver or spinach." Family members can then improve their diet based on this advice.

[0076] This system makes it possible to comprehensively manage the dietary habits of those requiring care, significantly reducing the burden on family members and caregivers.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[0080] Step 2:

[0081] Device: Sends the saved image data and date and time information to the server.

[0082] Step 3:

[0083] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[0084] Step 4:

[0085] Server: Records the analysis results in a database and saves the date and time of the meal, the type of ingredients, and the portion size in the database.

[0086] Step 5:

[0087] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[0088] Step 6:

[0089] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes consumption patterns and creates a list of the ingredients and quantities needed next time.

[0090] Step 7:

[0091] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[0092] Step 8:

[0093] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[0094] Step 9:

[0095] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[0096] Step 10:

[0097] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[0098] Step 11:

[0099] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[0100] Step 12:

[0101] Server: Evaluates nutritional balance and generates improvement suggestions if there are nutrient deficiencies or excesses. Notifies family members and caregivers of the suggestions.

[0102] Step 13:

[0103] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[0104] This allows for efficient management of the dietary habits of those requiring care, maintaining an appropriate nutritional balance while reducing the burden on family members and caregivers.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] Managing the daily eating habits of those requiring care is a significant burden for many family members and caregivers. It requires a wide range of tasks, including recording accurate dietary information, managing nutritional balance, and procuring necessary ingredients. There is a need for a support system that can reduce this management burden and help those requiring care maintain a healthy and balanced diet.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically placing orders with ingredient suppliers based on the ordering list, means for notifying the user of the ordering information, means for analyzing the accumulated meal data to evaluate the nutritional intake status, means for generating improvement suggestions if the nutritional balance is inappropriate, means for notifying the user of the suggestions, and means for analyzing past consumption patterns to predict future ingredient consumption, thereby enabling comprehensive management and support of the daily eating habits of the care recipient.

[0110] "Person requiring care" refers to an individual who requires ongoing care and support in their daily lives.

[0111] "Means for photographing meals" refers to a camera device or related equipment for recording meal contents as image data.

[0112] "Means for transmitting image data" refers to a communication means for transferring captured image data to a server.

[0113] "Means for analyzing image data to recognize the type and amount of ingredients" refers to algorithms or software that use image processing technology to identify the type and amount of ingredients.

[0114] "Means for recording the recognition results in a database" refers to methods and technologies for storing information such as the type and quantity of food in a database.

[0115] "Means for predicting the types and amounts of ingredients needed based on recorded dietary data" refers to algorithms and methods for analyzing past dietary data and predicting the types and amounts of ingredients needed in the future.

[0116] The "means for generating the next ingredient order list" refers to a mechanism for creating an order list based on the predicted types and amounts of ingredients needed.

[0117] The "means for automatically placing orders with ingredient suppliers" refers to a technology for automatically placing orders with ingredient suppliers using the generated order list.

[0118] "Means for notifying the user of order information" refers to methods and techniques for notifying the user of information regarding the order contents.

[0119] "Means for assessing nutritional intake status" refers to algorithms and methods for analyzing and assessing the intake status of individual nutrients from recorded dietary data.

[0120] "Means for generating improvement suggestions in the event of inadequate nutritional balance" refers to a system for detecting nutritional deficiencies and excesses and suggesting appropriate ingredients and dietary improvements.

[0121] "Means for notifying users of suggestions" refers to methods and techniques for notifying users of generated improvement suggestions.

[0122] "Means for predicting future food consumption by analyzing past consumption patterns" refers to algorithms and methods for analyzing past food consumption data and predicting future food consumption.

[0123] This invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are a device that takes photos of the food, a server that transmits and analyzes the image data, a database that records the recognition results and prediction results, and a means of communication with the user.

[0124] 1. Record and recognize food

[0125] Terminal

[0126] When a user eats, a camera device attached to the device automatically takes pictures of the meal. For example, a Raspberry Pi camera module is used. The captured image data is stored on the device along with date and time information, and then sent to a server via Wi-Fi or Bluetooth.

[0127] server

[0128] The server analyzes the received image data using generative AI models such as Google Cloud Vision API and TensorFlow to recognize the type and quantity of ingredients. The recognition results are recorded in a database such as MySQL. For detailed recognition, image analysis algorithms are run to detect and classify specific ingredients and their quantities.

[0129] For example, if a user has bread, salad, and coffee for breakfast, the device takes a photo of the breakfast and sends it along with the date and time to the server. The server then uses the Google Cloud Vision API to analyze the image and recognize specific ingredient information, such as "one piece of bread, one plate of salad, and one cup of coffee," and stores it in a database.

[0130] 2. Predicting necessary ingredients

[0131] server

[0132] The server analyzes past consumption patterns based on the accumulated meal records. Using Python and the pandas library, it predicts the ingredients needed and their quantities based on the analysis results. The next food ordering list is generated based on the predicted consumption amounts.

[0133] For example, if the server recognizes from the food record for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and create a list of the ingredients needed.

[0134] 3.Automatic ordering of ingredients

[0135] server

[0136] The server uses cloud services such as Amazon Web Services (AWS) to automatically place orders with partner food suppliers based on the generated order list. The order details include the ingredient name, quantity, and delivery address. The order details are notified to the user's smartphone or computer. Notifications are sent using Google Firebase.

[0137] User

[0138] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0139] For example, the server generates an ingredient list and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from an ingredient supplier. The order details are sent to the user's smartphone, and the user can check the details and make adjustments if necessary.

[0140] 4. Nutritional balance management

[0141] server

[0142] The server analyzes the nutritional intake status of the care recipient based on the recorded dietary data. It uses Python's Numpy and Scipy libraries to evaluate the nutrients in each ingredient and calculate the intake amount. Based on the results of the nutritional balance evaluation, if there is a nutrient deficiency or excess, a suggestion is made to add appropriate ingredients. This suggestion is then sent to the user's smartphone.

[0143] User

[0144] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0145] For example, if the past week's records show that a care recipient has an iron deficiency, the server will send a notification to the user's smartphone with advice such as "We recommend adding iron-rich foods such as liver and spinach." The user can then improve their diet based on this advice.

[0146] Prompt Sentence Examples

[0147] By inputting the following prompt sentences into the generative AI model, meal recognition and nutritional analysis functions can be realized.

[0148] "Please tell me the algorithm to analyze images of meals taken by care recipients and recognize the types and portions of ingredients."

[0149] "How can I predict what ingredients and amounts I'll need next week based on my diet data from the past week?"

[0150] "Please explain how to analyze recorded dietary data and generate suggestions for improving nutritional balance."

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Processing Steps

[0153] Step 1: Photograph your food

[0154] Terminal

[0155] A camera device attached to the device automatically takes pictures of the user's meal. Specifically, the image data is acquired using a Raspberry Pi camera module. The input is the image captured by the camera, and the output is the image data that is stored on the device.

[0156] Step 2: Sending image data

[0157] Terminal

[0158] The device sends the captured image data to the server via Wi-Fi or Bluetooth. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, an HTTP POST request is sent using the Python requests library.

[0159] Step 3: Image analysis

[0160] server

[0161] The server analyzes the image data using image analysis algorithms such as Google Cloud Vision API and TensorFlow. The input is the image data sent to the server, and the output is the analysis results. Specifically, the processing recognizes the types and quantities of ingredients contained in the image.

[0162] Step 4: Save the recognition results to a database

[0163] server

[0164] The server records the results of image analysis in a database such as MySQL. The input is the recognition results obtained by image analysis, and the output is a meal record stored in the database. Specifically, the recognition results are inserted into the database using an SQL query.

[0165] Step 5: Analyze consumption patterns

[0166] server

[0167] The server analyzes past consumption patterns based on the accumulated food record data. The input is the food records stored in the database, and the output is the analysis results of consumption patterns. Specifically, the past data is aggregated and analyzed using the Python pandas library.

[0168] Step 6: Generate an ingredient ordering list

[0169] server

[0170] The server generates the next ingredient order list based on the analysis results. The input is the result of consumption pattern analysis, and the output is an order list. Specifically, it lists the types and quantities of ingredients needed.

[0171] Step 7: Automated ordering of ingredients

[0172] server

[0173] The server automatically places orders with retailers based on the generated order list. The input is the order list, and the output is the execution result of the order. Specifically, the order procedure is carried out using AWS Lambda and API Gateway.

[0174] Step 8: Notification of order details

[0175] server

[0176] The server notifies the user of the order details. The input is the execution result of the order, and the output is the notification to the user. Specifically, the notification is sent using Google Firebase.

[0177] Step 9: Analyze nutritional status

[0178] server

[0179] The server analyzes the user's nutritional intake status based on the recorded dietary data. The input is the dietary records stored in the database, and the output is the evaluation results of the nutritional intake status. Specific processing involves using Python's Numpy and Scipy libraries to evaluate the nutrients in each food ingredient and calculate the intake amount.

[0180] Step 10: Suggestions for improving nutritional balance

[0181] server

[0182] The server evaluates nutritional balance and generates suggestions for adding appropriate ingredients to address nutrient deficiencies or excesses. The input is the assessment result of nutritional intake status, and the output is improvement suggestions. Specific operations include notifying the user of the generated suggestions.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] It is important to efficiently manage the daily dietary content of those requiring care and maintain an appropriate nutritional balance. However, current systems only partially address the dietary management of those requiring care and the automatic ordering of ingredients, making it difficult to comprehensively manage nutritional balance and resolve food shortages. Furthermore, no rational solutions are provided, including inventory management and automatic ordering at physical stores.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to the server, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from ingredient suppliers based on the ordering list, means for supporting ingredient management and inventory replenishment for the care recipient at a physical store, means for notifying the user of ordering information, means for analyzing the nutritional intake status of the care recipient, means for evaluating nutritional balance and generating improvement suggestions if inadequate, and means for analyzing past consumption patterns based on the care recipient's meal records to predict future consumption. This enables efficient dietary management of the care recipient, maintaining nutritional balance, and automatic replenishment of necessary ingredients.

[0188] "Care recipient" refers to an individual who requires assistance from others in daily living.

[0189] "Means for photographing meals" refers to a camera or image acquisition device for photographing the meals consumed by the person requiring care.

[0190] "Means for transmitting image data to a server" refers to means for sending image data of photographed food to a server via the Internet or other communication means.

[0191] "Means for analyzing image data" refers to algorithms or software that analyze captured image data and recognize the type and quantity of ingredients.

[0192] "Means for recording in a database" refers to a database management system that stores the analyzed data and allows it to be referenced later.

[0193] "Means for predicting the types and amounts of ingredients needed" refers to a system for predicting the types and amounts of ingredients needed in the future based on recorded past dietary data.

[0194] "Means for generating an ingredient order list" refers to software or algorithms that automatically generate the next order list based on the predicted ingredient needs and quantities.

[0195] The "means for automatically placing orders with food ingredient suppliers" refers to a means for automatically placing orders for ingredients with affiliated food ingredient suppliers based on the generated order list.

[0196] "A means to support food ingredient management and inventory replenishment for people requiring care in physical stores" refers to a system that uses the meal records and consumption patterns of people requiring care to streamline food ingredient inventory management and replenishment in physical stores.

[0197] The "means for notifying the user of order information" refers to a means for notifying the user of the generated order list and information related to the order contents.

[0198] "Means for analyzing nutritional intake status" refers to software or algorithms for analyzing the types and amounts of nutrients being ingested based on the dietary data of the person requiring care.

[0199] "Means for assessing nutritional balance and proposing improvements if inappropriate" refers to a system for assessing nutritional balance based on nutritional intake status and generating specific improvement proposals to supplement missing nutrients.

[0200] "Means for analyzing past consumption patterns to predict future consumption" refers to algorithms and software that analyze consumption patterns from recorded dietary data and predict future consumption.

[0201] The present invention provides a system for efficiently managing the daily dietary content of a care-requiring person and automating food ingredient management and inventory replenishment in a physical store while maintaining appropriate nutritional balance. Specific embodiments for implementing the present invention are described below.

[0202] 1. System Configuration

[0203] The system mainly consists of the following components:

[0204] A camera device for taking photos of meals for care recipients

[0205] A communication terminal for transmitting image data to a server

[0206] Software that analyzes image data to recognize the type and quantity of ingredients

[0207] A database management system for recording analysis results in a database

[0208] An algorithm that predicts the ingredients and amounts needed based on recorded meal data

[0209] Software for generating next food ordering list

[0210] A system to support food management and inventory replenishment in brick-and-mortar stores

[0211] Communication methods to notify users of ordering information and offers

[0212] Nutritional status analysis software used to analyze results

[0213] Software for assessing nutritional balance and generating improvement suggestions

[0214] Algorithms for analyzing past consumption patterns and predicting future consumption

[0215] 2. Program Overview

[0216] Hardware and Software Usage:

[0217] Hardware: Smartphones, camera devices

[0218] Software: Python, OpenCV, Requests, Web server, Database (e.g. MySQL)

[0219] Process flow:

[0220] 1. Image capture:

[0221] The terminal uses a camera device to take an image of the meal when the person requiring care eats the meal.

[0222] 2. Sending images:

[0223] The captured image data is sent via the device to a server via the Internet, allowing the server to receive information about the meal in real time.

[0224] 3. Image analysis and recognition:

[0225] The server analyzes the image data using Python and OpenCV to identify the type and quantity of ingredients, and the analyzed data is stored in a database.

[0226] 4. Forecast and order ingredients:

[0227] The server runs an algorithm (e.g., a machine learning model) that predicts the type and amount of ingredients needed next based on the recorded meal data. Based on the results, it generates a list of ingredients to order next and notifies the physical store.

[0228] 5. Inventory management in physical stores:

[0229] The physical store manages inventory based on the received order list and replenishes supplies to ensure necessary ingredients are available.

[0230] 6. User Notification and Nutrition Management:

[0231] The server notifies the user of the order information and the analysis results of the nutritional intake status. If the nutritional balance is inappropriate, the server generates improvement suggestions and sends them to the user.

[0232] 3. Specific Examples

[0233] If the server recognizes that bread, salad, and coffee are the breakfast items for a care recipient, it analyzes the consumption patterns of these items based on past data and automatically places an order with a physical store to replenish any missing ingredients.

[0234] Example prompt for a generative AI model:

[0235] "Please analyze the following food image and tell me the types and amounts of ingredients contained in it."

[0236] This system will improve the accuracy of dietary management for those requiring care and also streamline inventory management of food ingredients in physical stores.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The device uses a camera device to capture images of the meal while the care recipient is eating, and the captured images are given date and time information.

[0240] Input: Meal image, date and time information

[0241] Output: Image file of the photographed meal, date and time information

[0242] Step 2:

[0243] The terminal transmits the captured image data to a server using a communication means, and the image data is uploaded to the server via the Internet.

[0244] Input: Meal image file, date and time information

[0245] Output: Image file of the meal sent to the server, date and time information

[0246] Step 3:

[0247] The server analyzes the received image data using Python and OpenCV. This analysis identifies the type and quantity of ingredients. The analysis results include the identified type of ingredients, quantity, and date and time information.

[0248] Input: Image file of the meal sent to the server, date and time information

[0249] Data processing: Applying image analysis algorithms to identify ingredient types and quantities

[0250] Output: Identified ingredients, quantity, date and time

[0251] Step 4:

[0252] The server records the analysis results in a database, which stores the date and time of the meal, the type of ingredients identified, and the portion size.

[0253] Input: Identified ingredient type, quantity, date and time information

[0254] Data processing: Convert to database format

[0255] Output: Meal data stored in a database

[0256] Step 5:

[0257] The server predicts the ingredients needed and their quantities based on dietary data stored in a database, and the prediction is made by analyzing past consumption patterns.

[0258] Input: Food data stored in a database

[0259] Data Computing: Predicting future consumption using machine learning models

[0260] Output: Predicted ingredients and quantities needed

[0261] Step 6:

[0262] The server generates an order list for the next meal based on the predictions, which includes the types and quantities of ingredients needed.

[0263] Input: Predicted ingredients and quantities needed

[0264] Data processing: Convert to order list format

[0265] Output: Order list

[0266] Step 7:

[0267] The server automatically places orders with partner food suppliers based on the generated order list. The order details include the name, quantity, and delivery address of the required ingredients.

[0268] Input: Purchase Order List

[0269] Data calculation: Apply automatic ordering algorithm

[0270] Output: Order notification to ingredient supplier

[0271] Step 8:

[0272] The server notifies the user of the order information, who can then check the order details via a device such as a smartphone and adjust the order quantity if necessary.

[0273] Input: Generated purchase order list, purchase order notification

[0274] Output: User notification, confirmation result

[0275] Step 9:

[0276] The server analyzes the nutritional intake status of the care recipient, evaluates the nutrients ingested based on the dietary data, and generates appropriate improvement suggestions if there are any deficiencies.

[0277] Input: Food data stored in a database

[0278] Data calculation: Applying nutrient analysis algorithms

[0279] Output: Nutritional status report, improvement suggestions

[0280] Step 10:

[0281] The server then notifies the user of the nutritional improvement suggestions it has generated, allowing the user to adjust their diet based on this information.

[0282] Input: Nutritional status report, improvement suggestions

[0283] Output: User notification, adjustment results

[0284] As a concrete example, if a care recipient has bread, salad, and coffee for breakfast, the device takes a photo of it and sends it to the server. The server analyzes the image, recognizes the types and amounts of ingredients, and records them in a database. It then predicts the amount of ingredients needed based on past meal data, generates an order list for the physical store, places the order, and notifies the user. The server also provides information based on prompts such as, "Analyze the following meal image and tell me the types and amounts of ingredients it contains."

[0285] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0286] The present invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management, and also combines this with an emotion engine that recognizes the user's emotions. The main components of this system are as follows:

[0287] 1. Record and recognize food

[0288] Terminal

[0289] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[0290] The terminal transmits the stored image data to the server.

[0291] server

[0292] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[0293] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0294] Specific examples

[0295] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[0296] 2. Predicting necessary ingredients

[0297] server

[0298] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[0299] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[0300] Specific examples

[0301] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[0302] 3.Automatic ordering of ingredients

[0303] server

[0304] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[0305] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[0306] User

[0307] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0308] Specific examples

[0309] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[0310] 4. Nutritional balance management

[0311] server

[0312] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[0313] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[0314] The generated nutritional balance improvement proposal is notified to the user.

[0315] User

[0316] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0317] Specific examples

[0318] If the past week's records show that the care recipient has an iron deficiency, the server generates advice such as "We recommend adding iron-rich foods such as liver and spinach," and sends it to the family's smartphone. Family members can then improve their diet based on this advice.

[0319] 5. Combination of Emotion Engines

[0320] server

[0321] The server incorporates an emotion engine for recognizing the user's emotions, which analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0322] It runs algorithms that tailor orders and nutritional suggestions based on perceived emotions, for example, suggesting foods with a relaxing effect if the user is feeling stressed.

[0323] User

[0324] The notification method is also adjusted depending on the user's emotional state: if the user is feeling anxious, for example, the notification will be delivered in a calmer manner.

[0325] Specific examples

[0326] If the user is feeling stressed while confirming their order, the emotion engine will recognize their emotions and suggest adding relaxing herbal tea to the order. The notification will also be delivered in a calm tone, helping to reduce the user's stress.

[0327] This system comprehensively manages the diet of those requiring care, maintaining an appropriate nutritional balance while taking into account the user's emotional state, significantly reducing the burden on family members and caregivers.

[0328] The processing flow will be explained below.

[0329] Step 1:

[0330] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[0331] Step 2:

[0332] Device: Sends the saved image data and date and time information to the server.

[0333] Step 3:

[0334] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[0335] Step 4:

[0336] Server: Records the analysis results in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0337] Step 5:

[0338] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[0339] Step 6:

[0340] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes past consumption patterns and creates a list of the ingredients and quantities needed next time.

[0341] Step 7:

[0342] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[0343] Step 8:

[0344] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[0345] Step 9:

[0346] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[0347] Step 10:

[0348] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[0349] Step 11:

[0350] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[0351] Step 12:

[0352] Server: Based on the results of the nutritional balance assessment, if there is a nutrient deficiency or excess, it generates improvement suggestions and notifies the family or caregiver of the suggestions.

[0353] Step 13:

[0354] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[0355] Step 14:

[0356] Device: Uses a camera and microphone to capture your facial expressions and tone of voice as you review your meal and order details.

[0357] Step 15:

[0358] Device: Sends the captured data to the emotion engine, which analyzes the user's emotional state and sends the results to the server.

[0359] Step 16:

[0360] Server: Adjusts the order and nutritional suggestions based on the emotional state received from the emotion engine. For example, if the user is feeling stressed, it will suggest adding ingredients that have a relaxing effect.

[0361] Step 17:

[0362] Server: Notifies family members or caregivers with adjusted orders and nutritional suggestions. Notifications are also tailored to the user's emotional state. For example, if the user is feeling anxious, the notification will be delivered in a calming manner.

[0363] Step 18:

[0364] User: Family or caregiver reviews the order and nutrition suggestions provided and makes adjustments as needed.

[0365] Through these specific processing steps, the diet of the person requiring care can be efficiently managed, maintaining an appropriate nutritional balance while taking into consideration the user's emotional state, significantly reducing the burden on family members and caregivers.

[0366] Example 2

[0367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] While daily dietary management and nutritional balance for those requiring care are extremely important, they often place a significant burden on family members and caregivers. Furthermore, because it is necessary to consider the emotional state of the care recipient, a system that can do this efficiently is needed. While existing systems partially cover automatic ordering of ingredients and nutritional balance management, they lack functionality that takes the user's emotional state into account, and are therefore still insufficient. Therefore, there is a need to provide a system that can solve these issues and more effectively manage the diet of those requiring care while also taking the user's emotional state into consideration.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0370] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to an information processing device, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a storage device, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from suppliers based on the ordering list, means for notifying the user of the ordering information, means for recognizing the user's emotional state, and means for adjusting the order content and notification method based on the user's emotional state. This not only enables efficient dietary management and maintenance of nutritional balance for the care recipient, but also enables appropriate responses that take the user's emotional state into consideration.

[0371] "Persons requiring care" refers to people who have difficulty managing some or all of their daily lives on their own and require care and support.

[0372] "Meals" refers to the food and drink consumed daily by the person requiring care.

[0373] "Means for taking photographs" refers to a device for recording images of the meal, such as a camera device or a robot.

[0374] "Image data" refers to image information of the photographed meal.

[0375] "Information processing device" refers to a device for transmitting, receiving, and analyzing data, such as a server or computer system.

[0376] "Means for analysis" refers to the algorithms and software used to analyze image data and recognize the type and quantity of ingredients.

[0377] "Ingredients" refer to the individual food elements in a meal.

[0378] "Type" refers to the classification of multiple ingredients that are recognized.

[0379] "Quantity" refers to the quantity or volume of each ingredient.

[0380] "Recognition results" refers to information on the type and quantity of ingredients obtained from the analyzed image data.

[0381] "Storage device" refers to a device that stores information, such as a database or recording medium.

[0382] "Necessary ingredients" refers to foods that the care recipient is expected to consume in future meals.

[0383] "Order List" refers to a list of predicted ingredients and their quantities.

[0384] "Supplier" refers to a company that provides ingredients based on an order list.

[0385] "Means for automated ordering" refers to a system or process for automatically placing orders with suppliers based on an order list.

[0386] "Order information" refers to the type and quantity of ingredients ordered, delivery address information, etc.

[0387] "Means of notification" refers to the methods and techniques used to convey information to users.

[0388] "Emotional state" refers to the current mental and emotional state of the user or care recipient.

[0389] "Means of recognition" refers to technologies and systems for detecting and analyzing emotional states.

[0390] "Adjusting means" refers to a process or device for changing order content or notification methods based on emotional state.

[0391] This invention is a system that recognizes the daily dietary contents of a care recipient, automatically orders the necessary ingredients, and supports nutritional management, and also combines it with an emotion engine that recognizes the user's emotions. The main components of this system are three entities: a terminal, a server, and a user.

[0392] First, the device has a means to take pictures of the meal when the care recipient eats. Specifically, the pictures are taken using a robot or a camera device. The captured image data is saved on the device along with date and time information, and the saved data is sent to the server after undergoing preprocessing. This preprocessing includes resizing and normalizing the image.

[0393] The server then analyzes the received image data to identify the type and quantity of ingredients. The image analysis algorithm used here includes, for example, an image classification model using TensorFlow. The ingredient information obtained as a result of the recognition is stored in a database, which is a storage device. The stored information includes the type of ingredient, the quantity, and the date and time of the meal.

[0394] The server also has a way to predict what ingredients will be needed and in what quantities, based on the accumulated meal records. For this analysis, it uses the machine learning library Scikit-learn, for example, to predict future consumption based on past consumption patterns. It then generates the next ingredient ordering list based on the prediction results.

[0395] The server also includes a means for automatically sending the generated order list to suppliers. For example, using the Twilio API, an order request is sent along with information on the ingredients, quantity, and delivery address. The details of the order are notified to the user, who can check them via an application on their smartphone or computer. The user can then check the order and make adjustments as necessary. The adjusted information is then sent back to the server.

[0396] The server also has analytical tools for managing the nutritional balance of care recipients. Based on the recorded dietary data, it evaluates the nutrients in each ingredient using, for example, Pandas, and calculates the intake amount. Based on the results of the nutritional balance evaluation, it also generates suggestions for adding appropriate ingredients. These suggestions are also notified to the user.

[0397] The system also incorporates an emotion engine that recognizes the user's emotional state. The server runs emotion recognition algorithms, using Microsoft Azure's Emotion API to analyze the user's facial expressions and tone of voice, for example. It also includes a means to adjust order content and notification methods based on the recognized emotional state. For example, if it determines that the user is feeling stressed, an algorithm will be run to suggest foods with a relaxing effect.

[0398] Specific examples

[0399] As a concrete example, suppose a user takes a photo of a care recipient's breakfast. The image is taken by the device and sent to the server along with the date and time information. The server analyzes the image, recognizes it as "one piece of bread, one plate of salad, and one cup of coffee," and saves it in a database. Based on the past week's meal records, the server predicts the next meal's necessary ingredients: seven pieces of bread, two bags of salad vegetables, and one bag of coffee, and places an order with the supplier. The order details are notified to the user's smartphone, and the user can review the order and make adjustments if necessary. If the server determines that the care recipient is iron deficient, it will generate advice such as "it is recommended to add liver or spinach" and notify the user. In addition, if the server recognizes that the user is feeling stressed while confirming the order details, it will add a suggestion of relaxing herbal tea.

[0400] Prompt Sentence Examples

[0401] "Please describe the process flow for a system that takes photos of the meals of a care recipient and automatically orders ingredients for the next meal and manages nutrition. Please also explain how the system can be combined with an emotion engine to take the user's emotional state into account."

[0402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0403] Step 1: Photograph your meal and send the data

[0404] Device:

[0405] When the care recipient begins to eat, the camera device takes an image of the meal.

[0406] Input: Image from camera device

[0407] Data processing: resizing and normalizing images

[0408] Output: Preprocessed image

[0409] Specific operation: When the care recipient eats breakfast, the camera automatically takes a photo, and the image is saved on the device along with the date and time information. The pre-processed image is then sent to the server.

[0410] Step 2: Recognizing ingredients through image analysis

[0411] server:

[0412] The received image data is analyzed to recognize the type and quantity of ingredients.

[0413] Input: Preprocessed image data

[0414] Data computation: Analysis using image classification models with TensorFlow

[0415] Output: Data on the type and quantity of recognized ingredients

[0416] Specific operation: The server analyzes the received image, recognizes specific ingredient information such as "1 piece of bread, 1 plate of salad, 1 cup of coffee," and stores it in a database.

[0417] Step 3: Save your meal data

[0418] server:

[0419] The analyzed results are saved in a database.

[0420] Input: Recognized ingredient type and quantity

[0421] Data processing: Converting data into a format suitable for database format

[0422] Output: Success / failure of saving to database

[0423] Specific operation: The recognized ingredient information is saved in a database as "meal date and time, type of ingredient, and quantity."

[0424] Step 4: Predict the ingredients you need

[0425] server:

[0426] Based on accumulated meal data, it predicts the ingredients and amounts needed next time.

[0427] Input: Food data stored in the database

[0428] Data Computing: Analyzing historical consumption patterns with machine learning models using Scikit-learn

[0429] Output: Next food order list

[0430] Specific operation: The server analyzes the food data from the past week and predicts that "next week you will need 7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee."

[0431] Step 5: Automated ordering of ingredients

[0432] server:

[0433] Based on the predicted order list, orders are automatically placed with suppliers.

[0434] Input: Next food order list

[0435] Data processing: Convert the order list into an API request format

[0436] Output: Success / failure of sending purchase order request to supplier

[0437] Specific operation: The server places an order for "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" with the supplier's API, and the order details are notified to the user.

[0438] Step 6: User confirms and adjusts order

[0439] User:

[0440] Review orders and make adjustments as necessary.

[0441] Input: Order Notification

[0442] Data processing: Sending adjusted order information

[0443] Output: Success / failure of sending adjusted order information to the server

[0444] Specific operation: The user checks the order details on their smartphone or computer, adjusts the order quantity if necessary, and sends the adjustment information back to the server.

[0445] Step 7: Nutritional Balance

[0446] server:

[0447] Based on the recorded dietary data, nutritional intake status is analyzed and nutritional balance is evaluated.

[0448] Input: Food data stored in the database

[0449] Data calculation: Nutrient evaluation and calculation using the Pandas library

[0450] Output: Nutritional balance assessment results and improvement suggestions

[0451] Specific operation: If the user is found to be iron deficient based on the records from the past week, advice such as "It is recommended to add liver or spinach" is generated and notified to the user.

[0452] Step 8: Recognize your emotional state

[0453] server:

[0454] Analyze the user's emotional state and adjust notifications and order content.

[0455] Input: User facial and voice data

[0456] Data Computing: Sentiment Analysis using Microsoft Azure's Emotion API

[0457] Output: Perceived emotional state

[0458] What it does: If the user is feeling stressed while reviewing their order, the server will recognize this and add a suggestion for a relaxing herbal tea.

[0459] These steps will enable the system to manage the diet of those in need of care and maintain nutritional balance, while also taking into account the emotional state of the user.

[0460] (Application example 2)

[0461] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] In an aging society, managing the diet of those requiring care is a significant burden for families and caregivers. Maintaining nutritional balance and properly ordering ingredients is time-consuming, making consistent management difficult. Furthermore, there is a need to understand the emotional state of those requiring care and adjust ingredients and notifications accordingly, but currently this is often done manually and is inefficient.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of the nursing care recipient's mealtime behavior, a means for transmitting the captured image information to the information management device, a means for analyzing the image information to recognize the type and quantity of food, a means for saving the recognition results in a recording device, a means for predicting the food items and their quantities based on the recorded lunch data, a means for generating a next food order list based on the prediction, a means for automatically placing orders with a food provider based on the order list, a means for notifying the user of the order information, a means for recognizing the user's emotional state, and a means for adjusting the order content and notification method based on the recognized emotion. This automates the nursing care recipient's dietary management, enabling proper maintenance of nutritional balance and efficient ordering of ingredients. Furthermore, the server automatically suggests ingredients and adjusts the notification method based on the user's emotional state, reducing the burden of caregiving.

[0464] A "person requiring care" is an elderly person or a disabled person who requires assistance from others in daily life.

[0465] "Food-feeding behavior" refers to the act and state of consuming food.

[0466] "Means for taking photographs" refers to a method of recording the meal as images or videos using a device such as a camera.

[0467] "Image information" refers to data of captured images or videos.

[0468] An "information management device" refers to a server or cloud service that receives and stores captured image information.

[0469] The "means of analysis" refers to a method of identifying the type and quantity of ingredients from image information using artificial intelligence or image recognition algorithms.

[0470] A "recording device" refers to a database or storage system for saving analysis results.

[0471] "Meal data" refers to recorded information regarding the dietary content of a care recipient.

[0472] "Prediction methods" are methods for calculating the types and quantities of food that will be needed in the future based on past school lunch data.

[0473] "Order List" means a list of the types and quantities of food items that are expected to be needed.

[0474] "Food provider" means a supplier that provides food needs based on a forecast.

[0475] "Means for automatically ordering" means the method by which the system automatically orders the required food from the food provider.

[0476] "Means of notification" refers to the method used to notify users of order information and system suggestions. This can primarily be electronic means such as smartphones or email.

[0477] "Means for recognizing emotional states" refers to algorithms or devices that analyze facial expressions and voice data of the care recipient and their caregiver to identify emotions.

[0478] "Adjustment means" are methods for changing the content of suggestions or notification methods based on the perceived emotional state.

[0479] This system automates the dietary management of people requiring care, maintaining nutritional balance and efficiently ordering ingredients. The system also recognizes the user's emotional state and adjusts ingredient suggestions and notifications accordingly.

[0480] 1. Required Hardware and Software

[0481] Hardware:

[0482] Smartphone with camera: Used to take photos of the mealtime behavior of the care recipient

[0483] Server: Analyzes and stores data, and manages orders

[0484] software:

[0485] OpenCV: A library for image capture and processing

[0486] EmotionRecognition: A library for emotion analysis using facial and voice data

[0487] Flask / Django: Frameworks for implementing server-side APIs

[0488] 2. Detailed System Configuration

[0489] a. Photographing feeding behavior and sending data

[0490] The smartphone camera is used to take pictures of the person receiving care eating. This image information is sent from the smartphone to a server. A dedicated application is installed on the smartphone, automating the process of taking pictures and sending the images.

[0491] b. Image analysis and food ingredient recognition

[0492] The server analyzes the received image data and identifies the type and quantity of ingredients. This analysis uses an image recognition algorithm incorporating a deep learning model. The results are stored in a database and later used for evaluating nutritional balance and ordering food.

[0493] c. Forecasting food needs and generating an ordering list

[0494] The server uses past meal data to calculate the type and quantity of food needed for the next meal. Based on the prediction, a next order list is generated. This order list is automatically sent to the food provider.

[0495] d. Evaluation of nutritional balance and suggestions for improvement

[0496] The server analyzes the nutritional intake status of the care recipient based on the recorded meal data. If necessary, suggestions for improving nutritional balance are generated and notified to the user. Notifications are mainly sent via smartphone.

[0497] e. Recognizing and regulating emotional states

[0498] The server uses the EmotionRecognition library to recognize the emotional state of the care recipient and caregiver. Order content and notification methods are adjusted based on the emotional state. For example, if the user is feeling stressed, ingredients with a relaxing effect are suggested.

[0499] 3. Specific examples of use and prompts

[0500] Specific use cases

[0501] When a care recipient eats breakfast, a smartphone camera takes a photo of them eating, and the data is sent to a server. The server analyzes the image to identify the bread and salad, and the results are saved in a database. Using past data, the amount of bread and salad vegetables needed for the next week is predicted, and an order is automatically placed with the food provider. At this time, the system also recognizes that the user's emotional state is anxious, and suggests ordering more herbal tea, which has a relaxing effect.

[0502] Prompt Sentence Examples

[0503] "Analyze and save an image of a care recipient's breakfast (bread and salad) and generate a food list. Also, if the user's emotional state is recognized as anxious, suggest additional foods that have a relaxing effect."

[0504] This system highly automates the dietary management of those in need of care, reducing the burden on family members and caregivers while ensuring nutritional balance and enabling flexible responses according to the emotional state of the individual.

[0505] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0506] Step 1:

[0507] A smartphone device takes a photo of the meal of a care recipient. The device's camera captures the image and saves it along with the date and time information. The input is the image data acquired from the smartphone camera, and the output is the image and date and time information saved in the device.

[0508] Step 2:

[0509] The device sends the stored image data to the server. The input is the image data and date and time information stored in the device, and the output is the transmission of this data to the server. This transmission is done through a dedicated application.

[0510] Step 3:

[0511] The server analyzes the received image data and uses an image analysis algorithm (e.g., a deep learning model) to recognize the type and quantity of ingredients. The input is the image data sent to the server, and the output is the recognized type and quantity of ingredients.

[0512] Step 4:

[0513] The server stores the recognition results in a database. The input is the recognized food type and quantity data, and the output is a record stored in the database. This allows past meal records to be accumulated.

[0514] Step 5:

[0515] The server predicts the food needs and their quantities based on the accumulated food record data. It analyzes past consumption patterns and calculates the types and quantities of food needed next time. The input is the past food records accumulated in the database, and the output is the predicted food needs and their quantities.

[0516] Step 6:

[0517] The server generates the next food order list based on the prediction results. The input is the predicted data of food items and their quantities, and the output is the order list. The list includes the type of food and the required quantity.

[0518] Step 7:

[0519] The server automatically places orders with food providers based on the order list. The input is the generated order list, and the output is the order information sent to the food provider. Here, the order details include the food name, quantity, and delivery address.

[0520] Step 8:

[0521] The server notifies the user of the order information. The input is the details of the food ordered, and the output is the order information sent to the user's smartphone. The user can review this information and make adjustments if necessary.

[0522] Step 9:

[0523] The server analyzes the nutritional intake status of the care recipient. Based on the recorded meal data, the server evaluates the nutrients in each food item and calculates the intake amount. The input is the meal data recorded in the database, and the output is the evaluation result of nutritional balance.

[0524] Step 10:

[0525] The server generates improvement suggestions based on the nutritional balance assessment results if there are nutrient deficiencies or excesses. The input is the nutritional balance assessment results, and the output is improvement suggestions for nutritional balance. These suggestions are made in the form of adding appropriate foods.

[0526] Step 11:

[0527] The server recognizes the user's emotional state. Using a dedicated emotion analysis algorithm (EmotionRecognition library), it analyzes the user's facial expressions and tone of voice to identify the emotional state. The input is the user's facial expressions and voice data, and the output is the identified emotional state.

[0528] Step 12:

[0529] The server adjusts the order content and notification method based on the recognized emotional state. For example, if the user is feeling stressed, it will suggest foods with a relaxing effect and notify the user in a calm tone. The input is the identified emotional state and the analysis results described above, and the output is the adjusted order content and notification method.

[0530] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0532] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0533] [Second embodiment]

[0534] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0535] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0536] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0537] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0538] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0539] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0540] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0541] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0544] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0545] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0546] The present invention is a system that recognizes the daily dietary habits of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are as follows:

[0547] 1. Record and recognize food

[0548] Terminal

[0549] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[0550] The terminal transmits the stored image data to the server.

[0551] server

[0552] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[0553] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0554] Specific examples

[0555] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[0556] 2. Predicting necessary ingredients

[0557] server

[0558] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[0559] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[0560] Specific examples

[0561] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[0562] 3.Automatic ordering of ingredients

[0563] server

[0564] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[0565] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[0566] User

[0567] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0568] Specific examples

[0569] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[0570] 4. Nutritional balance management

[0571] server

[0572] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[0573] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[0574] The generated nutritional balance improvement proposal is notified to the user.

[0575] User

[0576] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0577] Specific examples

[0578] If the past week's records show that the care recipient has an iron deficiency, the server will send advice to the family member's smartphone, such as "We recommend adding iron-rich foods such as liver or spinach." Family members can then improve their diet based on this advice.

[0579] This system makes it possible to comprehensively manage the dietary habits of those requiring care, significantly reducing the burden on family members and caregivers.

[0580] The processing flow will be explained below.

[0581] Step 1:

[0582] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[0583] Step 2:

[0584] Device: Sends the saved image data and date and time information to the server.

[0585] Step 3:

[0586] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[0587] Step 4:

[0588] Server: Records the analysis results in a database and saves the date and time of the meal, the type of ingredients, and the portion size in the database.

[0589] Step 5:

[0590] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[0591] Step 6:

[0592] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes consumption patterns and creates a list of the ingredients and quantities needed next time.

[0593] Step 7:

[0594] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[0595] Step 8:

[0596] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[0597] Step 9:

[0598] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[0599] Step 10:

[0600] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[0601] Step 11:

[0602] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[0603] Step 12:

[0604] Server: Evaluates nutritional balance and generates improvement suggestions if there are nutrient deficiencies or excesses. Notifies family members and caregivers of the suggestions.

[0605] Step 13:

[0606] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[0607] This allows for efficient management of the dietary habits of those requiring care, maintaining an appropriate nutritional balance while reducing the burden on family members and caregivers.

[0608] Example 1

[0609] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] Managing the daily eating habits of those requiring care is a significant burden for many family members and caregivers. It requires a wide range of tasks, including recording accurate dietary information, managing nutritional balance, and procuring necessary ingredients. There is a need for a support system that can reduce this management burden and help those requiring care maintain a healthy and balanced diet.

[0611] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0612] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically placing orders with ingredient suppliers based on the ordering list, means for notifying the user of the ordering information, means for analyzing the accumulated meal data to evaluate the nutritional intake status, means for generating improvement suggestions if the nutritional balance is inappropriate, means for notifying the user of the suggestions, and means for analyzing past consumption patterns to predict future ingredient consumption, thereby enabling comprehensive management and support of the daily eating habits of the care recipient.

[0613] "Person requiring care" refers to an individual who requires ongoing care and support in their daily lives.

[0614] "Means for photographing meals" refers to a camera device or related equipment for recording meal contents as image data.

[0615] "Means for transmitting image data" refers to a communication means for transferring captured image data to a server.

[0616] "Means for analyzing image data to recognize the type and amount of ingredients" refers to algorithms or software that use image processing technology to identify the type and amount of ingredients.

[0617] "Means for recording the recognition results in a database" refers to methods and technologies for storing information such as the type and quantity of food in a database.

[0618] "Means for predicting the types and amounts of ingredients needed based on recorded dietary data" refers to algorithms and methods for analyzing past dietary data and predicting the types and amounts of ingredients needed in the future.

[0619] The "means for generating the next ingredient order list" refers to a mechanism for creating an order list based on the predicted types and amounts of ingredients needed.

[0620] The "means for automatically placing orders with ingredient suppliers" refers to a technology for automatically placing orders with ingredient suppliers using the generated order list.

[0621] "Means for notifying the user of order information" refers to methods and techniques for notifying the user of information regarding the order contents.

[0622] "Means for assessing nutritional intake status" refers to algorithms and methods for analyzing and assessing the intake status of individual nutrients from recorded dietary data.

[0623] "Means for generating improvement suggestions in the event of inadequate nutritional balance" refers to a system for detecting nutritional deficiencies and excesses and suggesting appropriate ingredients and dietary improvements.

[0624] "Means for notifying users of suggestions" refers to methods and techniques for notifying users of generated improvement suggestions.

[0625] "Means for predicting future food consumption by analyzing past consumption patterns" refers to algorithms and methods for analyzing past food consumption data and predicting future food consumption.

[0626] This invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are a device that takes photos of the food, a server that transmits and analyzes the image data, a database that records the recognition results and prediction results, and a means of communication with the user.

[0627] 1. Record and recognize food

[0628] Terminal

[0629] When a user eats, a camera device attached to the device automatically takes pictures of the meal. For example, a Raspberry Pi camera module is used. The captured image data is stored on the device along with date and time information, and then sent to a server via Wi-Fi or Bluetooth.

[0630] server

[0631] The server analyzes the received image data using generative AI models such as Google Cloud Vision API and TensorFlow to recognize the type and quantity of ingredients. The recognition results are recorded in a database such as MySQL. For detailed recognition, image analysis algorithms are run to detect and classify specific ingredients and their quantities.

[0632] For example, if a user has bread, salad, and coffee for breakfast, the device takes a photo of the breakfast and sends it along with the date and time to the server. The server then uses the Google Cloud Vision API to analyze the image and recognize specific ingredient information, such as "one piece of bread, one plate of salad, and one cup of coffee," and stores it in a database.

[0633] 2. Predicting necessary ingredients

[0634] server

[0635] The server analyzes past consumption patterns based on the accumulated meal records. Using Python and the pandas library, it predicts the ingredients needed and their quantities based on the analysis results. The next food ordering list is generated based on the predicted consumption amounts.

[0636] For example, if the server recognizes from the food record for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and create a list of the ingredients needed.

[0637] 3.Automatic ordering of ingredients

[0638] server

[0639] The server uses cloud services such as Amazon Web Services (AWS) to automatically place orders with partner food suppliers based on the generated order list. The order details include the ingredient name, quantity, and delivery address. The order details are notified to the user's smartphone or computer. Notifications are sent using Google Firebase.

[0640] User

[0641] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0642] For example, the server generates an ingredient list and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from an ingredient supplier. The order details are sent to the user's smartphone, and the user can check the details and make adjustments if necessary.

[0643] 4. Nutritional balance management

[0644] server

[0645] The server analyzes the nutritional intake status of the care recipient based on the recorded dietary data. It uses Python's Numpy and Scipy libraries to evaluate the nutrients in each ingredient and calculate the intake amount. Based on the results of the nutritional balance evaluation, if there is a nutrient deficiency or excess, a suggestion is made to add appropriate ingredients. This suggestion is then sent to the user's smartphone.

[0646] User

[0647] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0648] For example, if the past week's records show that a care recipient has an iron deficiency, the server will send a notification to the user's smartphone with advice such as "We recommend adding iron-rich foods such as liver and spinach." The user can then improve their diet based on this advice.

[0649] Prompt Sentence Examples

[0650] By inputting the following prompt sentences into the generative AI model, meal recognition and nutritional analysis functions can be realized.

[0651] "Please tell me the algorithm to analyze images of meals taken by care recipients and recognize the types and portions of ingredients."

[0652] "How can I predict what ingredients and amounts I'll need next week based on my diet data from the past week?"

[0653] "Please explain how to analyze recorded dietary data and generate suggestions for improving nutritional balance."

[0654] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0655] Processing Steps

[0656] Step 1: Photograph your food

[0657] Terminal

[0658] A camera device attached to the device automatically takes pictures of the user's meal. Specifically, the image data is acquired using a Raspberry Pi camera module. The input is the image captured by the camera, and the output is the image data that is stored on the device.

[0659] Step 2: Sending image data

[0660] Terminal

[0661] The device sends the captured image data to the server via Wi-Fi or Bluetooth. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, an HTTP POST request is sent using the Python requests library.

[0662] Step 3: Image analysis

[0663] server

[0664] The server analyzes the image data using image analysis algorithms such as Google Cloud Vision API and TensorFlow. The input is the image data sent to the server, and the output is the analysis results. Specifically, the processing recognizes the types and quantities of ingredients contained in the image.

[0665] Step 4: Save the recognition results to a database

[0666] server

[0667] The server records the results of image analysis in a database such as MySQL. The input is the recognition results obtained by image analysis, and the output is a meal record stored in the database. Specifically, the recognition results are inserted into the database using an SQL query.

[0668] Step 5: Analyze consumption patterns

[0669] server

[0670] The server analyzes past consumption patterns based on the accumulated food record data. The input is the food records stored in the database, and the output is the analysis results of consumption patterns. Specifically, the past data is aggregated and analyzed using the Python pandas library.

[0671] Step 6: Generate an ingredient ordering list

[0672] server

[0673] The server generates the next ingredient order list based on the analysis results. The input is the result of consumption pattern analysis, and the output is an order list. Specifically, it lists the types and quantities of ingredients needed.

[0674] Step 7: Automated ordering of ingredients

[0675] server

[0676] The server automatically places orders with retailers based on the generated order list. The input is the order list, and the output is the execution result of the order. Specifically, the order procedure is carried out using AWS Lambda and API Gateway.

[0677] Step 8: Notification of order details

[0678] server

[0679] The server notifies the user of the order details. The input is the execution result of the order, and the output is the notification to the user. Specifically, the notification is sent using Google Firebase.

[0680] Step 9: Analyze nutritional status

[0681] server

[0682] The server analyzes the user's nutritional intake status based on the recorded dietary data. The input is the dietary records stored in the database, and the output is the evaluation results of the nutritional intake status. Specific processing involves using Python's Numpy and Scipy libraries to evaluate the nutrients in each food ingredient and calculate the intake amount.

[0683] Step 10: Suggestions for improving nutritional balance

[0684] server

[0685] The server evaluates nutritional balance and generates suggestions for adding appropriate ingredients to address nutrient deficiencies or excesses. The input is the assessment result of nutritional intake status, and the output is improvement suggestions. Specific operations include notifying the user of the generated suggestions.

[0686] (Application example 1)

[0687] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] It is important to efficiently manage the daily dietary content of those requiring care and maintain an appropriate nutritional balance. However, current systems only partially address the dietary management of those requiring care and the automatic ordering of ingredients, making it difficult to comprehensively manage nutritional balance and resolve food shortages. Furthermore, no rational solutions are provided, including inventory management and automatic ordering at physical stores.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0690] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to the server, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from ingredient suppliers based on the ordering list, means for supporting ingredient management and inventory replenishment for the care recipient at a physical store, means for notifying the user of ordering information, means for analyzing the nutritional intake status of the care recipient, means for evaluating nutritional balance and generating improvement suggestions if inadequate, and means for analyzing past consumption patterns based on the care recipient's meal records to predict future consumption. This enables efficient dietary management of the care recipient, maintaining nutritional balance, and automatic replenishment of necessary ingredients.

[0691] "Care recipient" refers to an individual who requires assistance from others in daily living.

[0692] "Means for photographing meals" refers to a camera or image acquisition device for photographing the meals consumed by the person requiring care.

[0693] "Means for transmitting image data to a server" refers to means for sending image data of photographed food to a server via the Internet or other communication means.

[0694] "Means for analyzing image data" refers to algorithms or software that analyze captured image data and recognize the type and quantity of ingredients.

[0695] "Means for recording in a database" refers to a database management system that stores the analyzed data and allows it to be referenced later.

[0696] "Means for predicting the types and amounts of ingredients needed" refers to a system for predicting the types and amounts of ingredients needed in the future based on recorded past dietary data.

[0697] "Means for generating an ingredient order list" refers to software or algorithms that automatically generate the next order list based on the predicted ingredient needs and quantities.

[0698] The "means for automatically placing orders with food ingredient suppliers" refers to a means for automatically placing orders for ingredients with affiliated food ingredient suppliers based on the generated order list.

[0699] "A means to support food ingredient management and inventory replenishment for people requiring care in physical stores" refers to a system that uses the meal records and consumption patterns of people requiring care to streamline food ingredient inventory management and replenishment in physical stores.

[0700] The "means for notifying the user of order information" refers to a means for notifying the user of the generated order list and information related to the order contents.

[0701] "Means for analyzing nutritional intake status" refers to software or algorithms for analyzing the types and amounts of nutrients being ingested based on the dietary data of the person requiring care.

[0702] "Means for assessing nutritional balance and proposing improvements if inappropriate" refers to a system for assessing nutritional balance based on nutritional intake status and generating specific improvement proposals to supplement missing nutrients.

[0703] "Means for analyzing past consumption patterns to predict future consumption" refers to algorithms and software that analyze consumption patterns from recorded dietary data and predict future consumption.

[0704] The present invention provides a system for efficiently managing the daily dietary content of a care-requiring person and automating food ingredient management and inventory replenishment in a physical store while maintaining appropriate nutritional balance. Specific embodiments for implementing the present invention are described below.

[0705] 1. System Configuration

[0706] The system mainly consists of the following components:

[0707] A camera device for taking photos of meals for care recipients

[0708] A communication terminal for transmitting image data to a server

[0709] Software that analyzes image data to recognize the type and quantity of ingredients

[0710] A database management system for recording analysis results in a database

[0711] An algorithm that predicts the ingredients and amounts needed based on recorded meal data

[0712] Software for generating next food ordering list

[0713] A system to support food management and inventory replenishment in brick-and-mortar stores

[0714] Communication methods to notify users of ordering information and offers

[0715] Nutritional status analysis software used to analyze results

[0716] Software for assessing nutritional balance and generating improvement suggestions

[0717] Algorithms for analyzing past consumption patterns and predicting future consumption

[0718] 2. Program Overview

[0719] Hardware and Software Usage:

[0720] Hardware: Smartphones, camera devices

[0721] Software: Python, OpenCV, Requests, Web server, Database (e.g. MySQL)

[0722] Process flow:

[0723] 1. Image capture:

[0724] The terminal uses a camera device to take an image of the meal when the person requiring care eats the meal.

[0725] 2. Sending images:

[0726] The captured image data is sent via the device to a server via the Internet, allowing the server to receive information about the meal in real time.

[0727] 3. Image analysis and recognition:

[0728] The server analyzes the image data using Python and OpenCV to identify the type and quantity of ingredients, and the analyzed data is stored in a database.

[0729] 4. Forecast and order ingredients:

[0730] The server runs an algorithm (e.g., a machine learning model) that predicts the type and amount of ingredients needed next based on the recorded meal data. Based on the results, it generates a list of ingredients to order next and notifies the physical store.

[0731] 5. Inventory management in physical stores:

[0732] The physical store manages inventory based on the received order list and replenishes supplies to ensure necessary ingredients are available.

[0733] 6. User Notification and Nutrition Management:

[0734] The server notifies the user of the order information and the analysis results of the nutritional intake status. If the nutritional balance is inappropriate, the server generates improvement suggestions and sends them to the user.

[0735] 3. Specific Examples

[0736] If the server recognizes that bread, salad, and coffee are the breakfast items for a care recipient, it analyzes the consumption patterns of these items based on past data and automatically places an order with a physical store to replenish any missing ingredients.

[0737] Example prompt for a generative AI model:

[0738] "Please analyze the following food image and tell me the types and amounts of ingredients contained in it."

[0739] This system will improve the accuracy of dietary management for those requiring care and also streamline inventory management of food ingredients in physical stores.

[0740] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0741] Step 1:

[0742] The device uses a camera device to capture images of the meal while the care recipient is eating, and the captured images are given date and time information.

[0743] Input: Meal image, date and time information

[0744] Output: Image file of the photographed meal, date and time information

[0745] Step 2:

[0746] The terminal transmits the captured image data to a server using a communication means, and the image data is uploaded to the server via the Internet.

[0747] Input: Meal image file, date and time information

[0748] Output: Image file of the meal sent to the server, date and time information

[0749] Step 3:

[0750] The server analyzes the received image data using Python and OpenCV. This analysis identifies the type and quantity of ingredients. The analysis results include the identified type of ingredients, quantity, and date and time information.

[0751] Input: Image file of the meal sent to the server, date and time information

[0752] Data processing: Applying image analysis algorithms to identify ingredient types and quantities

[0753] Output: Identified ingredients, quantity, date and time

[0754] Step 4:

[0755] The server records the analysis results in a database, which stores the date and time of the meal, the type of ingredients identified, and the portion size.

[0756] Input: Identified ingredient type, quantity, date and time information

[0757] Data processing: Convert to database format

[0758] Output: Meal data stored in a database

[0759] Step 5:

[0760] The server predicts the ingredients needed and their quantities based on dietary data stored in a database, and the prediction is made by analyzing past consumption patterns.

[0761] Input: Food data stored in a database

[0762] Data Computing: Predicting future consumption using machine learning models

[0763] Output: Predicted ingredients and quantities needed

[0764] Step 6:

[0765] The server generates an order list for the next meal based on the predictions, which includes the types and quantities of ingredients needed.

[0766] Input: Predicted ingredients and quantities needed

[0767] Data processing: Convert to order list format

[0768] Output: Order list

[0769] Step 7:

[0770] The server automatically places orders with partner food suppliers based on the generated order list. The order details include the name, quantity, and delivery address of the required ingredients.

[0771] Input: Purchase Order List

[0772] Data calculation: Apply automatic ordering algorithm

[0773] Output: Order notification to ingredient supplier

[0774] Step 8:

[0775] The server notifies the user of the order information, who can then check the order details via a device such as a smartphone and adjust the order quantity if necessary.

[0776] Input: Generated purchase order list, purchase order notification

[0777] Output: User notification, confirmation result

[0778] Step 9:

[0779] The server analyzes the nutritional intake status of the care recipient, evaluates the nutrients ingested based on the dietary data, and generates appropriate improvement suggestions if there are any deficiencies.

[0780] Input: Food data stored in a database

[0781] Data calculation: Applying nutrient analysis algorithms

[0782] Output: Nutritional status report, improvement suggestions

[0783] Step 10:

[0784] The server then notifies the user of the nutritional improvement suggestions it has generated, allowing the user to adjust their diet based on this information.

[0785] Input: Nutritional status report, improvement suggestions

[0786] Output: User notification, adjustment results

[0787] As a concrete example, if a care recipient has bread, salad, and coffee for breakfast, the device takes a photo of it and sends it to the server. The server analyzes the image, recognizes the types and amounts of ingredients, and records them in a database. It then predicts the amount of ingredients needed based on past meal data, generates an order list for the physical store, places the order, and notifies the user. The server also provides information based on prompts such as, "Analyze the following meal image and tell me the types and amounts of ingredients it contains."

[0788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0789] The present invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management, and also combines this with an emotion engine that recognizes the user's emotions. The main components of this system are as follows:

[0790] 1. Record and recognize food

[0791] Terminal

[0792] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[0793] The terminal transmits the stored image data to the server.

[0794] server

[0795] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[0796] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0797] Specific examples

[0798] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[0799] 2. Predicting necessary ingredients

[0800] server

[0801] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[0802] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[0803] Specific examples

[0804] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[0805] 3.Automatic ordering of ingredients

[0806] server

[0807] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[0808] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[0809] User

[0810] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[0811] Specific examples

[0812] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[0813] 4. Nutritional balance management

[0814] server

[0815] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[0816] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[0817] The generated nutritional balance improvement proposal is notified to the user.

[0818] User

[0819] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[0820] Specific examples

[0821] If the past week's records show that the care recipient has an iron deficiency, the server generates advice such as "We recommend adding iron-rich foods such as liver and spinach," and sends it to the family's smartphone. Family members can then improve their diet based on this advice.

[0822] 5. Combination of Emotion Engines

[0823] server

[0824] The server incorporates an emotion engine for recognizing the user's emotions, which analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0825] It runs algorithms that tailor orders and nutritional suggestions based on perceived emotions, for example, suggesting foods with a relaxing effect if the user is feeling stressed.

[0826] User

[0827] The notification method is also adjusted depending on the user's emotional state: if the user is feeling anxious, for example, the notification will be delivered in a calmer manner.

[0828] Specific examples

[0829] If the user is feeling stressed while confirming their order, the emotion engine will recognize their emotions and suggest adding relaxing herbal tea to the order. The notification will also be delivered in a calm tone, helping to reduce the user's stress.

[0830] This system comprehensively manages the diet of those requiring care, maintaining an appropriate nutritional balance while taking into account the user's emotional state, significantly reducing the burden on family members and caregivers.

[0831] The processing flow will be explained below.

[0832] Step 1:

[0833] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[0834] Step 2:

[0835] Device: Sends the saved image data and date and time information to the server.

[0836] Step 3:

[0837] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[0838] Step 4:

[0839] Server: Records the analysis results in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[0840] Step 5:

[0841] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[0842] Step 6:

[0843] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes past consumption patterns and creates a list of the ingredients and quantities needed next time.

[0844] Step 7:

[0845] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[0846] Step 8:

[0847] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[0848] Step 9:

[0849] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[0850] Step 10:

[0851] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[0852] Step 11:

[0853] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[0854] Step 12:

[0855] Server: Based on the results of the nutritional balance assessment, if there is a nutrient deficiency or excess, it generates improvement suggestions and notifies the family or caregiver of the suggestions.

[0856] Step 13:

[0857] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[0858] Step 14:

[0859] Device: Uses a camera and microphone to capture your facial expressions and tone of voice as you review your meal and order details.

[0860] Step 15:

[0861] Device: Sends the captured data to the emotion engine, which analyzes the user's emotional state and sends the results to the server.

[0862] Step 16:

[0863] Server: Adjusts the order and nutritional suggestions based on the emotional state received from the emotion engine. For example, if the user is feeling stressed, it will suggest adding ingredients that have a relaxing effect.

[0864] Step 17:

[0865] Server: Notifies family members or caregivers with adjusted orders and nutritional suggestions. Notifications are also tailored to the user's emotional state. For example, if the user is feeling anxious, the notification will be delivered in a calming manner.

[0866] Step 18:

[0867] User: Family or caregiver reviews the order and nutrition suggestions provided and makes adjustments as needed.

[0868] Through these specific processing steps, the diet of the person requiring care can be efficiently managed, maintaining an appropriate nutritional balance while taking into consideration the user's emotional state, significantly reducing the burden on family members and caregivers.

[0869] Example 2

[0870] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0871] While daily dietary management and nutritional balance for those requiring care are extremely important, they often place a significant burden on family members and caregivers. Furthermore, because it is necessary to consider the emotional state of the care recipient, a system that can do this efficiently is needed. While existing systems partially cover automatic ordering of ingredients and nutritional balance management, they lack functionality that takes the user's emotional state into account, and are therefore still insufficient. Therefore, there is a need to provide a system that can solve these issues and more effectively manage the diet of those requiring care while also taking the user's emotional state into consideration.

[0872] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0873] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to an information processing device, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a storage device, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from suppliers based on the ordering list, means for notifying the user of the ordering information, means for recognizing the user's emotional state, and means for adjusting the order content and notification method based on the user's emotional state. This not only enables efficient dietary management and maintenance of nutritional balance for the care recipient, but also enables appropriate responses that take the user's emotional state into consideration.

[0874] "Persons requiring care" refers to people who have difficulty managing some or all of their daily lives on their own and require care and support.

[0875] "Meals" refers to the food and drink consumed daily by the person requiring care.

[0876] "Means for taking photographs" refers to a device for recording images of the meal, such as a camera device or a robot.

[0877] "Image data" refers to image information of the photographed meal.

[0878] "Information processing device" refers to a device for transmitting, receiving, and analyzing data, such as a server or computer system.

[0879] "Means for analysis" refers to the algorithms and software used to analyze image data and recognize the type and quantity of ingredients.

[0880] "Ingredients" refer to the individual food elements in a meal.

[0881] "Type" refers to the classification of multiple ingredients that are recognized.

[0882] "Quantity" refers to the quantity or volume of each ingredient.

[0883] "Recognition results" refers to information on the type and quantity of ingredients obtained from the analyzed image data.

[0884] "Storage device" refers to a device that stores information, such as a database or recording medium.

[0885] "Necessary ingredients" refers to foods that the care recipient is expected to consume in future meals.

[0886] "Order List" refers to a list of predicted ingredients and their quantities.

[0887] "Supplier" refers to a company that provides ingredients based on an order list.

[0888] "Means for automated ordering" refers to a system or process for automatically placing orders with suppliers based on an order list.

[0889] "Order information" refers to the type and quantity of ingredients ordered, delivery address information, etc.

[0890] "Means of notification" refers to the methods and techniques used to convey information to users.

[0891] "Emotional state" refers to the current mental and emotional state of the user or care recipient.

[0892] "Means of recognition" refers to technologies and systems for detecting and analyzing emotional states.

[0893] "Adjusting means" refers to a process or device for changing order content or notification methods based on emotional state.

[0894] This invention is a system that recognizes the daily dietary contents of a care recipient, automatically orders the necessary ingredients, and supports nutritional management, and also combines it with an emotion engine that recognizes the user's emotions. The main components of this system are three entities: a terminal, a server, and a user.

[0895] First, the device has a means to take pictures of the meal when the care recipient eats. Specifically, the pictures are taken using a robot or a camera device. The captured image data is saved on the device along with date and time information, and the saved data is sent to the server after undergoing preprocessing. This preprocessing includes resizing and normalizing the image.

[0896] The server then analyzes the received image data to identify the type and quantity of ingredients. The image analysis algorithm used here includes, for example, an image classification model using TensorFlow. The ingredient information obtained as a result of the recognition is stored in a database, which is a storage device. The stored information includes the type of ingredient, the quantity, and the date and time of the meal.

[0897] The server also has a way to predict what ingredients will be needed and in what quantities, based on the accumulated meal records. For this analysis, it uses the machine learning library Scikit-learn, for example, to predict future consumption based on past consumption patterns. It then generates the next ingredient ordering list based on the prediction results.

[0898] The server also includes a means for automatically sending the generated order list to suppliers. For example, using the Twilio API, an order request is sent along with information on the ingredients, quantity, and delivery address. The details of the order are notified to the user, who can check them via an application on their smartphone or computer. The user can then check the order and make adjustments as necessary. The adjusted information is then sent back to the server.

[0899] The server also has analytical tools for managing the nutritional balance of care recipients. Based on the recorded dietary data, it evaluates the nutrients in each ingredient using, for example, Pandas, and calculates the intake amount. Based on the results of the nutritional balance evaluation, it also generates suggestions for adding appropriate ingredients. These suggestions are also notified to the user.

[0900] The system also incorporates an emotion engine that recognizes the user's emotional state. The server runs emotion recognition algorithms, using Microsoft Azure's Emotion API to analyze the user's facial expressions and tone of voice, for example. It also includes a means to adjust order content and notification methods based on the recognized emotional state. For example, if it determines that the user is feeling stressed, an algorithm will be run to suggest foods with a relaxing effect.

[0901] Specific examples

[0902] As a concrete example, suppose a user takes a photo of a care recipient's breakfast. The image is taken by the device and sent to the server along with the date and time information. The server analyzes the image, recognizes it as "one piece of bread, one plate of salad, and one cup of coffee," and saves it in a database. Based on the past week's meal records, the server predicts the next meal's necessary ingredients: seven pieces of bread, two bags of salad vegetables, and one bag of coffee, and places an order with the supplier. The order details are notified to the user's smartphone, and the user can review the order and make adjustments if necessary. If the server determines that the care recipient is iron deficient, it will generate advice such as "it is recommended to add liver or spinach" and notify the user. In addition, if the server recognizes that the user is feeling stressed while confirming the order details, it will add a suggestion of relaxing herbal tea.

[0903] Prompt Sentence Examples

[0904] "Please describe the process flow for a system that takes photos of the meals of a care recipient and automatically orders ingredients for the next meal and manages nutrition. Please also explain how the system can be combined with an emotion engine to take the user's emotional state into account."

[0905] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0906] Step 1: Photograph your meal and send the data

[0907] Device:

[0908] When the care recipient begins to eat, the camera device takes an image of the meal.

[0909] Input: Image from camera device

[0910] Data processing: resizing and normalizing images

[0911] Output: Preprocessed image

[0912] Specific operation: When the care recipient eats breakfast, the camera automatically takes a photo, and the image is saved on the device along with the date and time information. The pre-processed image is then sent to the server.

[0913] Step 2: Recognizing ingredients through image analysis

[0914] server:

[0915] The received image data is analyzed to recognize the type and quantity of ingredients.

[0916] Input: Preprocessed image data

[0917] Data computation: Analysis using image classification models with TensorFlow

[0918] Output: Data on the type and quantity of recognized ingredients

[0919] Specific operation: The server analyzes the received image, recognizes specific ingredient information such as "1 piece of bread, 1 plate of salad, 1 cup of coffee," and stores it in a database.

[0920] Step 3: Save your meal data

[0921] server:

[0922] The analyzed results are saved in a database.

[0923] Input: Recognized ingredient type and quantity

[0924] Data processing: Converting data into a format suitable for database format

[0925] Output: Success / failure of saving to database

[0926] Specific operation: The recognized ingredient information is saved in a database as "meal date and time, type of ingredient, and quantity."

[0927] Step 4: Predict the ingredients you need

[0928] server:

[0929] Based on accumulated meal data, it predicts the ingredients and amounts needed next time.

[0930] Input: Food data stored in the database

[0931] Data Computing: Analyzing historical consumption patterns with machine learning models using Scikit-learn

[0932] Output: Next food order list

[0933] Specific operation: The server analyzes the food data from the past week and predicts that "next week you will need 7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee."

[0934] Step 5: Automated ordering of ingredients

[0935] server:

[0936] Based on the predicted order list, orders are automatically placed with suppliers.

[0937] Input: Next food order list

[0938] Data processing: Convert the order list into an API request format

[0939] Output: Success / failure of sending purchase order request to supplier

[0940] Specific operation: The server places an order for "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" with the supplier's API, and the order details are notified to the user.

[0941] Step 6: User confirms and adjusts order

[0942] User:

[0943] Review orders and make adjustments as necessary.

[0944] Input: Order Notification

[0945] Data processing: Sending adjusted order information

[0946] Output: Success / failure of sending adjusted order information to the server

[0947] Specific operation: The user checks the order details on their smartphone or computer, adjusts the order quantity if necessary, and sends the adjustment information back to the server.

[0948] Step 7: Nutritional Balance

[0949] server:

[0950] Based on the recorded dietary data, nutritional intake status is analyzed and nutritional balance is evaluated.

[0951] Input: Food data stored in the database

[0952] Data calculation: Nutrient evaluation and calculation using the Pandas library

[0953] Output: Nutritional balance assessment results and improvement suggestions

[0954] Specific operation: If the user is found to be iron deficient based on the records from the past week, advice such as "It is recommended to add liver or spinach" is generated and notified to the user.

[0955] Step 8: Recognize your emotional state

[0956] server:

[0957] Analyze the user's emotional state and adjust notifications and order content.

[0958] Input: User facial and voice data

[0959] Data Computing: Sentiment Analysis using Microsoft Azure's Emotion API

[0960] Output: Perceived emotional state

[0961] What it does: If the user is feeling stressed while reviewing their order, the server will recognize this and add a suggestion for a relaxing herbal tea.

[0962] These steps will enable the system to manage the diet of those in need of care and maintain nutritional balance, while also taking into account the emotional state of the user.

[0963] (Application example 2)

[0964] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0965] In an aging society, managing the diet of those requiring care is a significant burden for families and caregivers. Maintaining nutritional balance and properly ordering ingredients is time-consuming, making consistent management difficult. Furthermore, there is a need to understand the emotional state of those requiring care and adjust ingredients and notifications accordingly, but currently this is often done manually and is inefficient.

[0966] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of the nursing care recipient's mealtime behavior, a means for transmitting the captured image information to the information management device, a means for analyzing the image information to recognize the type and quantity of food, a means for saving the recognition results in a recording device, a means for predicting the food items and their quantities based on the recorded lunch data, a means for generating a next food order list based on the prediction, a means for automatically placing orders with a food provider based on the order list, a means for notifying the user of the order information, a means for recognizing the user's emotional state, and a means for adjusting the order content and notification method based on the recognized emotion. This automates the nursing care recipient's dietary management, enabling proper maintenance of nutritional balance and efficient ordering of ingredients. Furthermore, the server automatically suggests ingredients and adjusts the notification method based on the user's emotional state, reducing the burden of caregiving.

[0967] A "person requiring care" is an elderly person or a disabled person who requires assistance from others in daily life.

[0968] "Food-feeding behavior" refers to the act and state of consuming food.

[0969] "Means for taking photographs" refers to a method of recording the meal as images or videos using a device such as a camera.

[0970] "Image information" refers to data of captured images or videos.

[0971] An "information management device" refers to a server or cloud service that receives and stores captured image information.

[0972] The "means of analysis" refers to a method of identifying the type and quantity of ingredients from image information using artificial intelligence or image recognition algorithms.

[0973] A "recording device" refers to a database or storage system for saving analysis results.

[0974] "Meal data" refers to recorded information regarding the dietary content of a care recipient.

[0975] "Prediction methods" are methods for calculating the types and quantities of food that will be needed in the future based on past school lunch data.

[0976] "Order List" means a list of the types and quantities of food items that are expected to be needed.

[0977] "Food provider" means a supplier that provides food needs based on a forecast.

[0978] "Means for automatically ordering" means the method by which the system automatically orders the required food from the food provider.

[0979] "Means of notification" refers to the method used to notify users of order information and system suggestions. This can primarily be electronic means such as smartphones or email.

[0980] "Means for recognizing emotional states" refers to algorithms or devices that analyze facial expressions and voice data of the care recipient and their caregiver to identify emotions.

[0981] "Adjustment means" are methods for changing the content of suggestions or notification methods based on the perceived emotional state.

[0982] This system automates the dietary management of people requiring care, maintaining nutritional balance and efficiently ordering ingredients. The system also recognizes the user's emotional state and adjusts ingredient suggestions and notifications accordingly.

[0983] 1. Required Hardware and Software

[0984] Hardware:

[0985] Smartphone with camera: Used to take photos of the mealtime behavior of the care recipient

[0986] Server: Analyzes and stores data, and manages orders

[0987] software:

[0988] OpenCV: A library for image capture and processing

[0989] EmotionRecognition: A library for emotion analysis using facial and voice data

[0990] Flask / Django: Frameworks for implementing server-side APIs

[0991] 2. Detailed System Configuration

[0992] a. Photographing feeding behavior and sending data

[0993] The smartphone camera is used to take pictures of the person receiving care eating. This image information is sent from the smartphone to a server. A dedicated application is installed on the smartphone, automating the process of taking pictures and sending the images.

[0994] b. Image analysis and food ingredient recognition

[0995] The server analyzes the received image data and identifies the type and quantity of ingredients. This analysis uses an image recognition algorithm incorporating a deep learning model. The results are stored in a database and later used for evaluating nutritional balance and ordering food.

[0996] c. Forecasting food needs and generating an ordering list

[0997] The server uses past meal data to calculate the type and quantity of food needed for the next meal. Based on the prediction, a next order list is generated. This order list is automatically sent to the food provider.

[0998] d. Evaluation of nutritional balance and suggestions for improvement

[0999] The server analyzes the nutritional intake status of the care recipient based on the recorded meal data. If necessary, suggestions for improving nutritional balance are generated and notified to the user. Notifications are mainly sent via smartphone.

[1000] e. Recognizing and regulating emotional states

[1001] The server uses the EmotionRecognition library to recognize the emotional state of the care recipient and caregiver. Order content and notification methods are adjusted based on the emotional state. For example, if the user is feeling stressed, ingredients with a relaxing effect are suggested.

[1002] 3. Specific examples of use and prompts

[1003] Specific use cases

[1004] When a care recipient eats breakfast, a smartphone camera takes a photo of them eating, and the data is sent to a server. The server analyzes the image to identify the bread and salad, and the results are saved in a database. Using past data, the amount of bread and salad vegetables needed for the next week is predicted, and an order is automatically placed with the food provider. At this time, the system also recognizes that the user's emotional state is anxious, and suggests ordering more herbal tea, which has a relaxing effect.

[1005] Prompt Sentence Examples

[1006] "Analyze and save an image of a care recipient's breakfast (bread and salad) and generate a food list. Also, if the user's emotional state is recognized as anxious, suggest additional foods that have a relaxing effect."

[1007] This system highly automates the dietary management of those in need of care, reducing the burden on family members and caregivers while ensuring nutritional balance and enabling flexible responses according to the emotional state of the individual.

[1008] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1009] Step 1:

[1010] A smartphone device takes a photo of the meal of a care recipient. The device's camera captures the image and saves it along with the date and time information. The input is the image data acquired from the smartphone camera, and the output is the image and date and time information saved in the device.

[1011] Step 2:

[1012] The device sends the stored image data to the server. The input is the image data and date and time information stored in the device, and the output is the transmission of this data to the server. This transmission is done through a dedicated application.

[1013] Step 3:

[1014] The server analyzes the received image data and uses an image analysis algorithm (e.g., a deep learning model) to recognize the type and quantity of ingredients. The input is the image data sent to the server, and the output is the recognized type and quantity of ingredients.

[1015] Step 4:

[1016] The server stores the recognition results in a database. The input is the recognized food type and quantity data, and the output is a record stored in the database. This allows past meal records to be accumulated.

[1017] Step 5:

[1018] The server predicts the food needs and their quantities based on the accumulated food record data. It analyzes past consumption patterns and calculates the types and quantities of food needed next time. The input is the past food records accumulated in the database, and the output is the predicted food needs and their quantities.

[1019] Step 6:

[1020] The server generates the next food order list based on the prediction results. The input is the predicted data of food items and their quantities, and the output is the order list. The list includes the type of food and the required quantity.

[1021] Step 7:

[1022] The server automatically places orders with food providers based on the order list. The input is the generated order list, and the output is the order information sent to the food provider. Here, the order details include the food name, quantity, and delivery address.

[1023] Step 8:

[1024] The server notifies the user of the order information. The input is the details of the food ordered, and the output is the order information sent to the user's smartphone. The user can review this information and make adjustments if necessary.

[1025] Step 9:

[1026] The server analyzes the nutritional intake status of the care recipient. Based on the recorded meal data, the server evaluates the nutrients in each food item and calculates the intake amount. The input is the meal data recorded in the database, and the output is the evaluation result of nutritional balance.

[1027] Step 10:

[1028] The server generates improvement suggestions based on the nutritional balance assessment results if there are nutrient deficiencies or excesses. The input is the nutritional balance assessment results, and the output is improvement suggestions for nutritional balance. These suggestions are made in the form of adding appropriate foods.

[1029] Step 11:

[1030] The server recognizes the user's emotional state. Using a dedicated emotion analysis algorithm (EmotionRecognition library), it analyzes the user's facial expressions and tone of voice to identify the emotional state. The input is the user's facial expressions and voice data, and the output is the identified emotional state.

[1031] Step 12:

[1032] The server adjusts the order content and notification method based on the recognized emotional state. For example, if the user is feeling stressed, it will suggest foods with a relaxing effect and notify the user in a calm tone. The input is the identified emotional state and the analysis results described above, and the output is the adjusted order content and notification method.

[1033] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1035] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1036] [Third embodiment]

[1037] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1038] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1040] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1041] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1042] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1044] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1045] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1047] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1048] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1049] The present invention is a system that recognizes the daily dietary habits of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are as follows:

[1050] 1. Record and recognize food

[1051] Terminal

[1052] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[1053] The terminal transmits the stored image data to the server.

[1054] server

[1055] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[1056] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1057] Specific examples

[1058] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[1059] 2. Predicting necessary ingredients

[1060] server

[1061] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[1062] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[1063] Specific examples

[1064] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[1065] 3.Automatic ordering of ingredients

[1066] server

[1067] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[1068] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[1069] User

[1070] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1071] Specific examples

[1072] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[1073] 4. Nutritional balance management

[1074] server

[1075] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[1076] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[1077] The generated nutritional balance improvement proposal is notified to the user.

[1078] User

[1079] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1080] Specific examples

[1081] If the past week's records show that the care recipient has an iron deficiency, the server will send advice to the family member's smartphone, such as "We recommend adding iron-rich foods such as liver or spinach." Family members can then improve their diet based on this advice.

[1082] This system makes it possible to comprehensively manage the dietary habits of those requiring care, significantly reducing the burden on family members and caregivers.

[1083] The processing flow will be explained below.

[1084] Step 1:

[1085] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[1086] Step 2:

[1087] Device: Sends the saved image data and date and time information to the server.

[1088] Step 3:

[1089] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[1090] Step 4:

[1091] Server: Records the analysis results in a database and saves the date and time of the meal, the type of ingredients, and the portion size in the database.

[1092] Step 5:

[1093] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[1094] Step 6:

[1095] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes consumption patterns and creates a list of the ingredients and quantities needed next time.

[1096] Step 7:

[1097] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[1098] Step 8:

[1099] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[1100] Step 9:

[1101] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[1102] Step 10:

[1103] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[1104] Step 11:

[1105] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[1106] Step 12:

[1107] Server: Evaluates nutritional balance and generates improvement suggestions if there are nutrient deficiencies or excesses. Notifies family members and caregivers of the suggestions.

[1108] Step 13:

[1109] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[1110] This allows for efficient management of the dietary habits of those requiring care, maintaining an appropriate nutritional balance while reducing the burden on family members and caregivers.

[1111] Example 1

[1112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1113] Managing the daily eating habits of those requiring care is a significant burden for many family members and caregivers. It requires a wide range of tasks, including recording accurate dietary information, managing nutritional balance, and procuring necessary ingredients. There is a need for a support system that can reduce this management burden and help those requiring care maintain a healthy and balanced diet.

[1114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1115] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically placing orders with ingredient suppliers based on the ordering list, means for notifying the user of the ordering information, means for analyzing the accumulated meal data to evaluate the nutritional intake status, means for generating improvement suggestions if the nutritional balance is inappropriate, means for notifying the user of the suggestions, and means for analyzing past consumption patterns to predict future ingredient consumption, thereby enabling comprehensive management and support of the daily eating habits of the care recipient.

[1116] "Person requiring care" refers to an individual who requires ongoing care and support in their daily lives.

[1117] "Means for photographing meals" refers to a camera device or related equipment for recording meal contents as image data.

[1118] "Means for transmitting image data" refers to a communication means for transferring captured image data to a server.

[1119] "Means for analyzing image data to recognize the type and amount of ingredients" refers to algorithms or software that use image processing technology to identify the type and amount of ingredients.

[1120] "Means for recording the recognition results in a database" refers to methods and technologies for storing information such as the type and quantity of food in a database.

[1121] "Means for predicting the types and amounts of ingredients needed based on recorded dietary data" refers to algorithms and methods for analyzing past dietary data and predicting the types and amounts of ingredients needed in the future.

[1122] The "means for generating the next ingredient order list" refers to a mechanism for creating an order list based on the predicted types and amounts of ingredients needed.

[1123] The "means for automatically placing orders with ingredient suppliers" refers to a technology for automatically placing orders with ingredient suppliers using the generated order list.

[1124] "Means for notifying the user of order information" refers to methods and techniques for notifying the user of information regarding the order contents.

[1125] "Means for assessing nutritional intake status" refers to algorithms and methods for analyzing and assessing the intake status of individual nutrients from recorded dietary data.

[1126] "Means for generating improvement suggestions in the event of inadequate nutritional balance" refers to a system for detecting nutritional deficiencies and excesses and suggesting appropriate ingredients and dietary improvements.

[1127] "Means for notifying users of suggestions" refers to methods and techniques for notifying users of generated improvement suggestions.

[1128] "Means for predicting future food consumption by analyzing past consumption patterns" refers to algorithms and methods for analyzing past food consumption data and predicting future food consumption.

[1129] This invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are a device that takes photos of the food, a server that transmits and analyzes the image data, a database that records the recognition results and prediction results, and a means of communication with the user.

[1130] 1. Record and recognize food

[1131] Terminal

[1132] When a user eats, a camera device attached to the device automatically takes pictures of the meal. For example, a Raspberry Pi camera module is used. The captured image data is stored on the device along with date and time information, and then sent to a server via Wi-Fi or Bluetooth.

[1133] server

[1134] The server analyzes the received image data using generative AI models such as Google Cloud Vision API and TensorFlow to recognize the type and quantity of ingredients. The recognition results are recorded in a database such as MySQL. For detailed recognition, image analysis algorithms are run to detect and classify specific ingredients and their quantities.

[1135] For example, if a user has bread, salad, and coffee for breakfast, the device takes a photo of the breakfast and sends it along with the date and time to the server. The server then uses the Google Cloud Vision API to analyze the image and recognize specific ingredient information, such as "one piece of bread, one plate of salad, and one cup of coffee," and stores it in a database.

[1136] 2. Predicting necessary ingredients

[1137] server

[1138] The server analyzes past consumption patterns based on the accumulated meal records. Using Python and the pandas library, it predicts the ingredients needed and their quantities based on the analysis results. The next food ordering list is generated based on the predicted consumption amounts.

[1139] For example, if the server recognizes from the food record for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and create a list of the ingredients needed.

[1140] 3.Automatic ordering of ingredients

[1141] server

[1142] The server uses cloud services such as Amazon Web Services (AWS) to automatically place orders with partner food suppliers based on the generated order list. The order details include the ingredient name, quantity, and delivery address. The order details are notified to the user's smartphone or computer. Notifications are sent using Google Firebase.

[1143] User

[1144] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1145] For example, the server generates an ingredient list and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from an ingredient supplier. The order details are sent to the user's smartphone, and the user can check the details and make adjustments if necessary.

[1146] 4. Nutritional balance management

[1147] server

[1148] The server analyzes the nutritional intake status of the care recipient based on the recorded dietary data. It uses Python's Numpy and Scipy libraries to evaluate the nutrients in each ingredient and calculate the intake amount. Based on the results of the nutritional balance evaluation, if there is a nutrient deficiency or excess, a suggestion is made to add appropriate ingredients. This suggestion is then sent to the user's smartphone.

[1149] User

[1150] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1151] For example, if the past week's records show that a care recipient has an iron deficiency, the server will send a notification to the user's smartphone with advice such as "We recommend adding iron-rich foods such as liver and spinach." The user can then improve their diet based on this advice.

[1152] Prompt Sentence Examples

[1153] By inputting the following prompt sentences into the generative AI model, meal recognition and nutritional analysis functions can be realized.

[1154] "Please tell me the algorithm to analyze images of meals taken by care recipients and recognize the types and portions of ingredients."

[1155] "How can I predict what ingredients and amounts I'll need next week based on my diet data from the past week?"

[1156] "Please explain how to analyze recorded dietary data and generate suggestions for improving nutritional balance."

[1157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1158] Processing Steps

[1159] Step 1: Photograph your food

[1160] Terminal

[1161] A camera device attached to the device automatically takes pictures of the user's meal. Specifically, the image data is acquired using a Raspberry Pi camera module. The input is the image captured by the camera, and the output is the image data that is stored on the device.

[1162] Step 2: Sending image data

[1163] Terminal

[1164] The device sends the captured image data to the server via Wi-Fi or Bluetooth. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, an HTTP POST request is sent using the Python requests library.

[1165] Step 3: Image analysis

[1166] server

[1167] The server analyzes the image data using image analysis algorithms such as Google Cloud Vision API and TensorFlow. The input is the image data sent to the server, and the output is the analysis results. Specifically, the processing recognizes the types and quantities of ingredients contained in the image.

[1168] Step 4: Save the recognition results to a database

[1169] server

[1170] The server records the results of image analysis in a database such as MySQL. The input is the recognition results obtained by image analysis, and the output is a meal record stored in the database. Specifically, the recognition results are inserted into the database using an SQL query.

[1171] Step 5: Analyze consumption patterns

[1172] server

[1173] The server analyzes past consumption patterns based on the accumulated food record data. The input is the food records stored in the database, and the output is the analysis results of consumption patterns. Specifically, the past data is aggregated and analyzed using the Python pandas library.

[1174] Step 6: Generate an ingredient ordering list

[1175] server

[1176] The server generates the next ingredient order list based on the analysis results. The input is the result of consumption pattern analysis, and the output is an order list. Specifically, it lists the types and quantities of ingredients needed.

[1177] Step 7: Automated ordering of ingredients

[1178] server

[1179] The server automatically places orders with retailers based on the generated order list. The input is the order list, and the output is the execution result of the order. Specifically, the order procedure is carried out using AWS Lambda and API Gateway.

[1180] Step 8: Notification of order details

[1181] server

[1182] The server notifies the user of the order details. The input is the execution result of the order, and the output is the notification to the user. Specifically, the notification is sent using Google Firebase.

[1183] Step 9: Analyze nutritional status

[1184] server

[1185] The server analyzes the user's nutritional intake status based on the recorded dietary data. The input is the dietary records stored in the database, and the output is the evaluation results of the nutritional intake status. Specific processing involves using Python's Numpy and Scipy libraries to evaluate the nutrients in each food ingredient and calculate the intake amount.

[1186] Step 10: Suggestions for improving nutritional balance

[1187] server

[1188] The server evaluates nutritional balance and generates suggestions for adding appropriate ingredients to address nutrient deficiencies or excesses. The input is the assessment result of nutritional intake status, and the output is improvement suggestions. Specific operations include notifying the user of the generated suggestions.

[1189] (Application example 1)

[1190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] It is important to efficiently manage the daily dietary content of those requiring care and maintain an appropriate nutritional balance. However, current systems only partially address the dietary management of those requiring care and the automatic ordering of ingredients, making it difficult to comprehensively manage nutritional balance and resolve food shortages. Furthermore, no rational solutions are provided, including inventory management and automatic ordering at physical stores.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1193] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to the server, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from ingredient suppliers based on the ordering list, means for supporting ingredient management and inventory replenishment for the care recipient at a physical store, means for notifying the user of ordering information, means for analyzing the nutritional intake status of the care recipient, means for evaluating nutritional balance and generating improvement suggestions if inadequate, and means for analyzing past consumption patterns based on the care recipient's meal records to predict future consumption. This enables efficient dietary management of the care recipient, maintaining nutritional balance, and automatic replenishment of necessary ingredients.

[1194] "Care recipient" refers to an individual who requires assistance from others in daily living.

[1195] "Means for photographing meals" refers to a camera or image acquisition device for photographing the meals consumed by the person requiring care.

[1196] "Means for transmitting image data to a server" refers to means for sending image data of photographed food to a server via the Internet or other communication means.

[1197] "Means for analyzing image data" refers to algorithms or software that analyze captured image data and recognize the type and quantity of ingredients.

[1198] "Means for recording in a database" refers to a database management system that stores the analyzed data and allows it to be referenced later.

[1199] "Means for predicting the types and amounts of ingredients needed" refers to a system for predicting the types and amounts of ingredients needed in the future based on recorded past dietary data.

[1200] "Means for generating an ingredient order list" refers to software or algorithms that automatically generate the next order list based on the predicted ingredient needs and quantities.

[1201] The "means for automatically placing orders with food ingredient suppliers" refers to a means for automatically placing orders for ingredients with affiliated food ingredient suppliers based on the generated order list.

[1202] "A means to support food ingredient management and inventory replenishment for people requiring care in physical stores" refers to a system that uses the meal records and consumption patterns of people requiring care to streamline food ingredient inventory management and replenishment in physical stores.

[1203] The "means for notifying the user of order information" refers to a means for notifying the user of the generated order list and information related to the order contents.

[1204] "Means for analyzing nutritional intake status" refers to software or algorithms for analyzing the types and amounts of nutrients being ingested based on the dietary data of the person requiring care.

[1205] "Means for assessing nutritional balance and proposing improvements if inappropriate" refers to a system for assessing nutritional balance based on nutritional intake status and generating specific improvement proposals to supplement missing nutrients.

[1206] "Means for analyzing past consumption patterns to predict future consumption" refers to algorithms and software that analyze consumption patterns from recorded dietary data and predict future consumption.

[1207] The present invention provides a system for efficiently managing the daily dietary content of a care-requiring person and automating food ingredient management and inventory replenishment in a physical store while maintaining appropriate nutritional balance. Specific embodiments for implementing the present invention are described below.

[1208] 1. System Configuration

[1209] The system mainly consists of the following components:

[1210] A camera device for taking photos of meals for care recipients

[1211] A communication terminal for transmitting image data to a server

[1212] Software that analyzes image data to recognize the type and quantity of ingredients

[1213] A database management system for recording analysis results in a database

[1214] An algorithm that predicts the ingredients and amounts needed based on recorded meal data

[1215] Software for generating next food ordering list

[1216] A system to support food management and inventory replenishment in brick-and-mortar stores

[1217] Communication methods to notify users of ordering information and offers

[1218] Nutritional status analysis software used to analyze results

[1219] Software for assessing nutritional balance and generating improvement suggestions

[1220] Algorithms for analyzing past consumption patterns and predicting future consumption

[1221] 2. Program Overview

[1222] Hardware and Software Usage:

[1223] Hardware: Smartphones, camera devices

[1224] Software: Python, OpenCV, Requests, Web server, Database (e.g. MySQL)

[1225] Process flow:

[1226] 1. Image capture:

[1227] The terminal uses a camera device to take an image of the meal when the person requiring care eats the meal.

[1228] 2. Sending images:

[1229] The captured image data is sent via the device to a server via the Internet, allowing the server to receive information about the meal in real time.

[1230] 3. Image analysis and recognition:

[1231] The server analyzes the image data using Python and OpenCV to identify the type and quantity of ingredients, and the analyzed data is stored in a database.

[1232] 4. Forecast and order ingredients:

[1233] The server runs an algorithm (e.g., a machine learning model) that predicts the type and amount of ingredients needed next based on the recorded meal data. Based on the results, it generates a list of ingredients to order next and notifies the physical store.

[1234] 5. Inventory management in physical stores:

[1235] The physical store manages inventory based on the received order list and replenishes supplies to ensure necessary ingredients are available.

[1236] 6. User Notification and Nutrition Management:

[1237] The server notifies the user of the order information and the analysis results of the nutritional intake status. If the nutritional balance is inappropriate, the server generates improvement suggestions and sends them to the user.

[1238] 3. Specific Examples

[1239] If the server recognizes that bread, salad, and coffee are the breakfast items for a care recipient, it analyzes the consumption patterns of these items based on past data and automatically places an order with a physical store to replenish any missing ingredients.

[1240] Example prompt for a generative AI model:

[1241] "Please analyze the following food image and tell me the types and amounts of ingredients contained in it."

[1242] This system will improve the accuracy of dietary management for those requiring care and also streamline inventory management of food ingredients in physical stores.

[1243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1244] Step 1:

[1245] The device uses a camera device to capture images of the meal while the care recipient is eating, and the captured images are given date and time information.

[1246] Input: Meal image, date and time information

[1247] Output: Image file of the photographed meal, date and time information

[1248] Step 2:

[1249] The terminal transmits the captured image data to a server using a communication means, and the image data is uploaded to the server via the Internet.

[1250] Input: Meal image file, date and time information

[1251] Output: Image file of the meal sent to the server, date and time information

[1252] Step 3:

[1253] The server analyzes the received image data using Python and OpenCV. This analysis identifies the type and quantity of ingredients. The analysis results include the identified type of ingredients, quantity, and date and time information.

[1254] Input: Image file of the meal sent to the server, date and time information

[1255] Data processing: Applying image analysis algorithms to identify ingredient types and quantities

[1256] Output: Identified ingredients, quantity, date and time

[1257] Step 4:

[1258] The server records the analysis results in a database, which stores the date and time of the meal, the type of ingredients identified, and the portion size.

[1259] Input: Identified ingredient type, quantity, date and time information

[1260] Data processing: Convert to database format

[1261] Output: Meal data stored in a database

[1262] Step 5:

[1263] The server predicts the ingredients needed and their quantities based on dietary data stored in a database, and the prediction is made by analyzing past consumption patterns.

[1264] Input: Food data stored in a database

[1265] Data Computing: Predicting future consumption using machine learning models

[1266] Output: Predicted ingredients and quantities needed

[1267] Step 6:

[1268] The server generates an order list for the next meal based on the predictions, which includes the types and quantities of ingredients needed.

[1269] Input: Predicted ingredients and quantities needed

[1270] Data processing: Convert to order list format

[1271] Output: Order list

[1272] Step 7:

[1273] The server automatically places orders with partner food suppliers based on the generated order list. The order details include the name, quantity, and delivery address of the required ingredients.

[1274] Input: Purchase Order List

[1275] Data calculation: Apply automatic ordering algorithm

[1276] Output: Order notification to ingredient supplier

[1277] Step 8:

[1278] The server notifies the user of the order information, who can then check the order details via a device such as a smartphone and adjust the order quantity if necessary.

[1279] Input: Generated purchase order list, purchase order notification

[1280] Output: User notification, confirmation result

[1281] Step 9:

[1282] The server analyzes the nutritional intake status of the care recipient, evaluates the nutrients ingested based on the dietary data, and generates appropriate improvement suggestions if there are any deficiencies.

[1283] Input: Food data stored in a database

[1284] Data calculation: Applying nutrient analysis algorithms

[1285] Output: Nutritional status report, improvement suggestions

[1286] Step 10:

[1287] The server then notifies the user of the nutritional improvement suggestions it has generated, allowing the user to adjust their diet based on this information.

[1288] Input: Nutritional status report, improvement suggestions

[1289] Output: User notification, adjustment results

[1290] As a concrete example, if a care recipient has bread, salad, and coffee for breakfast, the device takes a photo of it and sends it to the server. The server analyzes the image, recognizes the types and amounts of ingredients, and records them in a database. It then predicts the amount of ingredients needed based on past meal data, generates an order list for the physical store, places the order, and notifies the user. The server also provides information based on prompts such as, "Analyze the following meal image and tell me the types and amounts of ingredients it contains."

[1291] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1292] The present invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management, and also combines this with an emotion engine that recognizes the user's emotions. The main components of this system are as follows:

[1293] 1. Record and recognize food

[1294] Terminal

[1295] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[1296] The terminal transmits the stored image data to the server.

[1297] server

[1298] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[1299] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1300] Specific examples

[1301] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[1302] 2. Predicting necessary ingredients

[1303] server

[1304] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[1305] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[1306] Specific examples

[1307] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[1308] 3.Automatic ordering of ingredients

[1309] server

[1310] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[1311] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[1312] User

[1313] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1314] Specific examples

[1315] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[1316] 4. Nutritional balance management

[1317] server

[1318] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[1319] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[1320] The generated nutritional balance improvement proposal is notified to the user.

[1321] User

[1322] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1323] Specific examples

[1324] If the past week's records show that the care recipient has an iron deficiency, the server generates advice such as "We recommend adding iron-rich foods such as liver and spinach," and sends it to the family's smartphone. Family members can then improve their diet based on this advice.

[1325] 5. Combination of Emotion Engines

[1326] server

[1327] The server incorporates an emotion engine for recognizing the user's emotions, which analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[1328] It runs algorithms that tailor orders and nutritional suggestions based on perceived emotions, for example, suggesting foods with a relaxing effect if the user is feeling stressed.

[1329] User

[1330] The notification method is also adjusted depending on the user's emotional state: if the user is feeling anxious, for example, the notification will be delivered in a calmer manner.

[1331] Specific examples

[1332] If the user is feeling stressed while confirming their order, the emotion engine will recognize their emotions and suggest adding relaxing herbal tea to the order. The notification will also be delivered in a calm tone, helping to reduce the user's stress.

[1333] This system comprehensively manages the diet of those requiring care, maintaining an appropriate nutritional balance while taking into account the user's emotional state, significantly reducing the burden on family members and caregivers.

[1334] The processing flow will be explained below.

[1335] Step 1:

[1336] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[1337] Step 2:

[1338] Device: Sends the saved image data and date and time information to the server.

[1339] Step 3:

[1340] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[1341] Step 4:

[1342] Server: Records the analysis results in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1343] Step 5:

[1344] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[1345] Step 6:

[1346] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes past consumption patterns and creates a list of the ingredients and quantities needed next time.

[1347] Step 7:

[1348] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[1349] Step 8:

[1350] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[1351] Step 9:

[1352] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[1353] Step 10:

[1354] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[1355] Step 11:

[1356] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[1357] Step 12:

[1358] Server: Based on the results of the nutritional balance assessment, if there is a nutrient deficiency or excess, it generates improvement suggestions and notifies the family or caregiver of the suggestions.

[1359] Step 13:

[1360] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[1361] Step 14:

[1362] Device: Uses a camera and microphone to capture your facial expressions and tone of voice as you review your meal and order details.

[1363] Step 15:

[1364] Device: Sends the captured data to the emotion engine, which analyzes the user's emotional state and sends the results to the server.

[1365] Step 16:

[1366] Server: Adjusts the order and nutritional suggestions based on the emotional state received from the emotion engine. For example, if the user is feeling stressed, it will suggest adding ingredients that have a relaxing effect.

[1367] Step 17:

[1368] Server: Notifies family members or caregivers with adjusted orders and nutritional suggestions. Notifications are also tailored to the user's emotional state. For example, if the user is feeling anxious, the notification will be delivered in a calming manner.

[1369] Step 18:

[1370] User: Family or caregiver reviews the order and nutrition suggestions provided and makes adjustments as needed.

[1371] Through these specific processing steps, the diet of the person requiring care can be efficiently managed, maintaining an appropriate nutritional balance while taking into consideration the user's emotional state, significantly reducing the burden on family members and caregivers.

[1372] Example 2

[1373] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1374] While daily dietary management and nutritional balance for those requiring care are extremely important, they often place a significant burden on family members and caregivers. Furthermore, because it is necessary to consider the emotional state of the care recipient, a system that can do this efficiently is needed. While existing systems partially cover automatic ordering of ingredients and nutritional balance management, they lack functionality that takes the user's emotional state into account, and are therefore still insufficient. Therefore, there is a need to provide a system that can solve these issues and more effectively manage the diet of those requiring care while also taking the user's emotional state into consideration.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1376] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to an information processing device, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a storage device, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from suppliers based on the ordering list, means for notifying the user of the ordering information, means for recognizing the user's emotional state, and means for adjusting the order content and notification method based on the user's emotional state. This not only enables efficient dietary management and maintenance of nutritional balance for the care recipient, but also enables appropriate responses that take the user's emotional state into consideration.

[1377] "Persons requiring care" refers to people who have difficulty managing some or all of their daily lives on their own and require care and support.

[1378] "Meals" refers to the food and drink consumed daily by the person requiring care.

[1379] "Means for taking photographs" refers to a device for recording images of the meal, such as a camera device or a robot.

[1380] "Image data" refers to image information of the photographed meal.

[1381] "Information processing device" refers to a device for transmitting, receiving, and analyzing data, such as a server or computer system.

[1382] "Means for analysis" refers to the algorithms and software used to analyze image data and recognize the type and quantity of ingredients.

[1383] "Ingredients" refer to the individual food elements in a meal.

[1384] "Type" refers to the classification of multiple ingredients that are recognized.

[1385] "Quantity" refers to the quantity or volume of each ingredient.

[1386] "Recognition results" refers to information on the type and quantity of ingredients obtained from the analyzed image data.

[1387] "Storage device" refers to a device that stores information, such as a database or recording medium.

[1388] "Necessary ingredients" refers to foods that the care recipient is expected to consume in future meals.

[1389] "Order List" refers to a list of predicted ingredients and their quantities.

[1390] "Supplier" refers to a company that provides ingredients based on an order list.

[1391] "Means for automated ordering" refers to a system or process for automatically placing orders with suppliers based on an order list.

[1392] "Order information" refers to the type and quantity of ingredients ordered, delivery address information, etc.

[1393] "Means of notification" refers to the methods and techniques used to convey information to users.

[1394] "Emotional state" refers to the current mental and emotional state of the user or care recipient.

[1395] "Means of recognition" refers to technologies and systems for detecting and analyzing emotional states.

[1396] "Adjusting means" refers to a process or device for changing order content or notification methods based on emotional state.

[1397] This invention is a system that recognizes the daily dietary contents of a care recipient, automatically orders the necessary ingredients, and supports nutritional management, and also combines it with an emotion engine that recognizes the user's emotions. The main components of this system are three entities: a terminal, a server, and a user.

[1398] First, the device has a means to take pictures of the meal when the care recipient eats. Specifically, the pictures are taken using a robot or a camera device. The captured image data is saved on the device along with date and time information, and the saved data is sent to the server after undergoing preprocessing. This preprocessing includes resizing and normalizing the image.

[1399] The server then analyzes the received image data to identify the type and quantity of ingredients. The image analysis algorithm used here includes, for example, an image classification model using TensorFlow. The ingredient information obtained as a result of the recognition is stored in a database, which is a storage device. The stored information includes the type of ingredient, the quantity, and the date and time of the meal.

[1400] The server also has a way to predict what ingredients will be needed and in what quantities, based on the accumulated meal records. For this analysis, it uses the machine learning library Scikit-learn, for example, to predict future consumption based on past consumption patterns. It then generates the next ingredient ordering list based on the prediction results.

[1401] The server also includes a means for automatically sending the generated order list to suppliers. For example, using the Twilio API, an order request is sent along with information on the ingredients, quantity, and delivery address. The details of the order are notified to the user, who can check them via an application on their smartphone or computer. The user can then check the order and make adjustments as necessary. The adjusted information is then sent back to the server.

[1402] The server also has analytical tools for managing the nutritional balance of care recipients. Based on the recorded dietary data, it evaluates the nutrients in each ingredient using, for example, Pandas, and calculates the intake amount. Based on the results of the nutritional balance evaluation, it also generates suggestions for adding appropriate ingredients. These suggestions are also notified to the user.

[1403] The system also incorporates an emotion engine that recognizes the user's emotional state. The server runs emotion recognition algorithms, using Microsoft Azure's Emotion API to analyze the user's facial expressions and tone of voice, for example. It also includes a means to adjust order content and notification methods based on the recognized emotional state. For example, if it determines that the user is feeling stressed, an algorithm will be run to suggest foods with a relaxing effect.

[1404] Specific examples

[1405] As a concrete example, suppose a user takes a photo of a care recipient's breakfast. The image is taken by the device and sent to the server along with the date and time information. The server analyzes the image, recognizes it as "one piece of bread, one plate of salad, and one cup of coffee," and saves it in a database. Based on the past week's meal records, the server predicts the next meal's necessary ingredients: seven pieces of bread, two bags of salad vegetables, and one bag of coffee, and places an order with the supplier. The order details are notified to the user's smartphone, and the user can review the order and make adjustments if necessary. If the server determines that the care recipient is iron deficient, it will generate advice such as "it is recommended to add liver or spinach" and notify the user. In addition, if the server recognizes that the user is feeling stressed while confirming the order details, it will add a suggestion of relaxing herbal tea.

[1406] Prompt Sentence Examples

[1407] "Please describe the process flow for a system that takes photos of the meals of a care recipient and automatically orders ingredients for the next meal and manages nutrition. Please also explain how the system can be combined with an emotion engine to take the user's emotional state into account."

[1408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1409] Step 1: Photograph your meal and send the data

[1410] Device:

[1411] When the care recipient begins to eat, the camera device takes an image of the meal.

[1412] Input: Image from camera device

[1413] Data processing: resizing and normalizing images

[1414] Output: Preprocessed image

[1415] Specific operation: When the care recipient eats breakfast, the camera automatically takes a photo, and the image is saved on the device along with the date and time information. The pre-processed image is then sent to the server.

[1416] Step 2: Recognizing ingredients through image analysis

[1417] server:

[1418] The received image data is analyzed to recognize the type and quantity of ingredients.

[1419] Input: Preprocessed image data

[1420] Data computation: Analysis using image classification models with TensorFlow

[1421] Output: Data on the type and quantity of recognized ingredients

[1422] Specific operation: The server analyzes the received image, recognizes specific ingredient information such as "1 piece of bread, 1 plate of salad, 1 cup of coffee," and stores it in a database.

[1423] Step 3: Save your meal data

[1424] server:

[1425] The analyzed results are saved in a database.

[1426] Input: Recognized ingredient type and quantity

[1427] Data processing: Converting data into a format suitable for database format

[1428] Output: Success / failure of saving to database

[1429] Specific operation: The recognized ingredient information is saved in a database as "meal date and time, type of ingredient, and quantity."

[1430] Step 4: Predict the ingredients you need

[1431] server:

[1432] Based on accumulated meal data, it predicts the ingredients and amounts needed next time.

[1433] Input: Food data stored in the database

[1434] Data Computing: Analyzing historical consumption patterns with machine learning models using Scikit-learn

[1435] Output: Next food order list

[1436] Specific operation: The server analyzes the food data from the past week and predicts that "next week you will need 7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee."

[1437] Step 5: Automated ordering of ingredients

[1438] server:

[1439] Based on the predicted order list, orders are automatically placed with suppliers.

[1440] Input: Next food order list

[1441] Data processing: Convert the order list into an API request format

[1442] Output: Success / failure of sending purchase order request to supplier

[1443] Specific operation: The server places an order for "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" with the supplier's API, and the order details are notified to the user.

[1444] Step 6: User confirms and adjusts order

[1445] User:

[1446] Review orders and make adjustments as necessary.

[1447] Input: Order Notification

[1448] Data processing: Sending adjusted order information

[1449] Output: Success / failure of sending adjusted order information to the server

[1450] Specific operation: The user checks the order details on their smartphone or computer, adjusts the order quantity if necessary, and sends the adjustment information back to the server.

[1451] Step 7: Nutritional Balance

[1452] server:

[1453] Based on the recorded dietary data, nutritional intake status is analyzed and nutritional balance is evaluated.

[1454] Input: Food data stored in the database

[1455] Data calculation: Nutrient evaluation and calculation using the Pandas library

[1456] Output: Nutritional balance assessment results and improvement suggestions

[1457] Specific operation: If the user is found to be iron deficient based on the records from the past week, advice such as "It is recommended to add liver or spinach" is generated and notified to the user.

[1458] Step 8: Recognize your emotional state

[1459] server:

[1460] Analyze the user's emotional state and adjust notifications and order content.

[1461] Input: User facial and voice data

[1462] Data Computing: Sentiment Analysis using Microsoft Azure's Emotion API

[1463] Output: Perceived emotional state

[1464] What it does: If the user is feeling stressed while reviewing their order, the server will recognize this and add a suggestion for a relaxing herbal tea.

[1465] These steps will enable the system to manage the diet of those in need of care and maintain nutritional balance, while also taking into account the emotional state of the user.

[1466] (Application example 2)

[1467] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1468] In an aging society, managing the diet of those requiring care is a significant burden for families and caregivers. Maintaining nutritional balance and properly ordering ingredients is time-consuming, making consistent management difficult. Furthermore, there is a need to understand the emotional state of those requiring care and adjust ingredients and notifications accordingly, but currently this is often done manually and is inefficient.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of the nursing care recipient's mealtime behavior, a means for transmitting the captured image information to the information management device, a means for analyzing the image information to recognize the type and quantity of food, a means for saving the recognition results in a recording device, a means for predicting the food items and their quantities based on the recorded lunch data, a means for generating a next food order list based on the prediction, a means for automatically placing orders with a food provider based on the order list, a means for notifying the user of the order information, a means for recognizing the user's emotional state, and a means for adjusting the order content and notification method based on the recognized emotion. This automates the nursing care recipient's dietary management, enabling proper maintenance of nutritional balance and efficient ordering of ingredients. Furthermore, the server automatically suggests ingredients and adjusts the notification method based on the user's emotional state, reducing the burden of caregiving.

[1470] A "person requiring care" is an elderly person or a disabled person who requires assistance from others in daily life.

[1471] "Food-feeding behavior" refers to the act and state of consuming food.

[1472] "Means for taking photographs" refers to a method of recording the meal as images or videos using a device such as a camera.

[1473] "Image information" refers to data of captured images or videos.

[1474] An "information management device" refers to a server or cloud service that receives and stores captured image information.

[1475] The "means of analysis" refers to a method of identifying the type and quantity of ingredients from image information using artificial intelligence or image recognition algorithms.

[1476] A "recording device" refers to a database or storage system for saving analysis results.

[1477] "Meal data" refers to recorded information regarding the dietary content of a care recipient.

[1478] "Prediction methods" are methods for calculating the types and quantities of food that will be needed in the future based on past school lunch data.

[1479] "Order List" means a list of the types and quantities of food items that are expected to be needed.

[1480] "Food provider" means a supplier that provides food needs based on a forecast.

[1481] "Means for automatically ordering" means the method by which the system automatically orders the required food from the food provider.

[1482] "Means of notification" refers to the method used to notify users of order information and system suggestions. This can primarily be electronic means such as smartphones or email.

[1483] "Means for recognizing emotional states" refers to algorithms or devices that analyze facial expressions and voice data of the care recipient and their caregiver to identify emotions.

[1484] "Adjustment means" are methods for changing the content of suggestions or notification methods based on the perceived emotional state.

[1485] This system automates the dietary management of people requiring care, maintaining nutritional balance and efficiently ordering ingredients. The system also recognizes the user's emotional state and adjusts ingredient suggestions and notifications accordingly.

[1486] 1. Required Hardware and Software

[1487] Hardware:

[1488] Smartphone with camera: Used to take photos of the mealtime behavior of the care recipient

[1489] Server: Analyzes and stores data, and manages orders

[1490] software:

[1491] OpenCV: A library for image capture and processing

[1492] EmotionRecognition: A library for emotion analysis using facial and voice data

[1493] Flask / Django: Frameworks for implementing server-side APIs

[1494] 2. Detailed System Configuration

[1495] a. Photographing feeding behavior and sending data

[1496] The smartphone camera is used to take pictures of the person receiving care eating. This image information is sent from the smartphone to a server. A dedicated application is installed on the smartphone, automating the process of taking pictures and sending the images.

[1497] b. Image analysis and food ingredient recognition

[1498] The server analyzes the received image data and identifies the type and quantity of ingredients. This analysis uses an image recognition algorithm incorporating a deep learning model. The results are stored in a database and later used for evaluating nutritional balance and ordering food.

[1499] c. Forecasting food needs and generating an ordering list

[1500] The server uses past meal data to calculate the type and quantity of food needed for the next meal. Based on the prediction, a next order list is generated. This order list is automatically sent to the food provider.

[1501] d. Evaluation of nutritional balance and suggestions for improvement

[1502] The server analyzes the nutritional intake status of the care recipient based on the recorded meal data. If necessary, suggestions for improving nutritional balance are generated and notified to the user. Notifications are mainly sent via smartphone.

[1503] e. Recognizing and regulating emotional states

[1504] The server uses the EmotionRecognition library to recognize the emotional state of the care recipient and caregiver. Order content and notification methods are adjusted based on the emotional state. For example, if the user is feeling stressed, ingredients with a relaxing effect are suggested.

[1505] 3. Specific examples of use and prompts

[1506] Specific use cases

[1507] When a care recipient eats breakfast, a smartphone camera takes a photo of them eating, and the data is sent to a server. The server analyzes the image to identify the bread and salad, and the results are saved in a database. Using past data, the amount of bread and salad vegetables needed for the next week is predicted, and an order is automatically placed with the food provider. At this time, the system also recognizes that the user's emotional state is anxious, and suggests ordering more herbal tea, which has a relaxing effect.

[1508] Prompt Sentence Examples

[1509] "Analyze and save an image of a care recipient's breakfast (bread and salad) and generate a food list. Also, if the user's emotional state is recognized as anxious, suggest additional foods that have a relaxing effect."

[1510] This system highly automates the dietary management of those in need of care, reducing the burden on family members and caregivers while ensuring nutritional balance and enabling flexible responses according to the emotional state of the individual.

[1511] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1512] Step 1:

[1513] A smartphone device takes a photo of the meal of a care recipient. The device's camera captures the image and saves it along with the date and time information. The input is the image data acquired from the smartphone camera, and the output is the image and date and time information saved in the device.

[1514] Step 2:

[1515] The device sends the stored image data to the server. The input is the image data and date and time information stored in the device, and the output is the transmission of this data to the server. This transmission is done through a dedicated application.

[1516] Step 3:

[1517] The server analyzes the received image data and uses an image analysis algorithm (e.g., a deep learning model) to recognize the type and quantity of ingredients. The input is the image data sent to the server, and the output is the recognized type and quantity of ingredients.

[1518] Step 4:

[1519] The server stores the recognition results in a database. The input is the recognized food type and quantity data, and the output is a record stored in the database. This allows past meal records to be accumulated.

[1520] Step 5:

[1521] The server predicts the food needs and their quantities based on the accumulated food record data. It analyzes past consumption patterns and calculates the types and quantities of food needed next time. The input is the past food records accumulated in the database, and the output is the predicted food needs and their quantities.

[1522] Step 6:

[1523] The server generates the next food order list based on the prediction results. The input is the predicted data of food items and their quantities, and the output is the order list. The list includes the type of food and the required quantity.

[1524] Step 7:

[1525] The server automatically places orders with food providers based on the order list. The input is the generated order list, and the output is the order information sent to the food provider. Here, the order details include the food name, quantity, and delivery address.

[1526] Step 8:

[1527] The server notifies the user of the order information. The input is the details of the food ordered, and the output is the order information sent to the user's smartphone. The user can review this information and make adjustments if necessary.

[1528] Step 9:

[1529] The server analyzes the nutritional intake status of the care recipient. Based on the recorded meal data, the server evaluates the nutrients in each food item and calculates the intake amount. The input is the meal data recorded in the database, and the output is the evaluation result of nutritional balance.

[1530] Step 10:

[1531] The server generates improvement suggestions based on the nutritional balance assessment results if there are nutrient deficiencies or excesses. The input is the nutritional balance assessment results, and the output is improvement suggestions for nutritional balance. These suggestions are made in the form of adding appropriate foods.

[1532] Step 11:

[1533] The server recognizes the user's emotional state. Using a dedicated emotion analysis algorithm (EmotionRecognition library), it analyzes the user's facial expressions and tone of voice to identify the emotional state. The input is the user's facial expressions and voice data, and the output is the identified emotional state.

[1534] Step 12:

[1535] The server adjusts the order content and notification method based on the recognized emotional state. For example, if the user is feeling stressed, it will suggest foods with a relaxing effect and notify the user in a calm tone. The input is the identified emotional state and the analysis results described above, and the output is the adjusted order content and notification method.

[1536] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1537] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1538] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1539] [Fourth embodiment]

[1540] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1541] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1542] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1543] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1544] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1545] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1546] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1547] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1548] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1549] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1550] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1551] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1552] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] The present invention is a system that recognizes the daily dietary habits of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are as follows:

[1554] 1. Record and recognize food

[1555] Terminal

[1556] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[1557] The terminal transmits the stored image data to the server.

[1558] server

[1559] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[1560] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1561] Specific examples

[1562] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[1563] 2. Predicting necessary ingredients

[1564] server

[1565] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[1566] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[1567] Specific examples

[1568] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[1569] 3.Automatic ordering of ingredients

[1570] server

[1571] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[1572] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[1573] User

[1574] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1575] Specific examples

[1576] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[1577] 4. Nutritional balance management

[1578] server

[1579] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[1580] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[1581] The generated nutritional balance improvement proposal is notified to the user.

[1582] User

[1583] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1584] Specific examples

[1585] If the past week's records show that the care recipient has an iron deficiency, the server will send advice to the family member's smartphone, such as "We recommend adding iron-rich foods such as liver or spinach." Family members can then improve their diet based on this advice.

[1586] This system makes it possible to comprehensively manage the dietary habits of those requiring care, significantly reducing the burden on family members and caregivers.

[1587] The processing flow will be explained below.

[1588] Step 1:

[1589] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[1590] Step 2:

[1591] Device: Sends the saved image data and date and time information to the server.

[1592] Step 3:

[1593] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[1594] Step 4:

[1595] Server: Records the analysis results in a database and saves the date and time of the meal, the type of ingredients, and the portion size in the database.

[1596] Step 5:

[1597] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[1598] Step 6:

[1599] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes consumption patterns and creates a list of the ingredients and quantities needed next time.

[1600] Step 7:

[1601] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[1602] Step 8:

[1603] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[1604] Step 9:

[1605] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[1606] Step 10:

[1607] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[1608] Step 11:

[1609] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[1610] Step 12:

[1611] Server: Evaluates nutritional balance and generates improvement suggestions if there are nutrient deficiencies or excesses. Notifies family members and caregivers of the suggestions.

[1612] Step 13:

[1613] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[1614] This allows for efficient management of the dietary habits of those requiring care, maintaining an appropriate nutritional balance while reducing the burden on family members and caregivers.

[1615] Example 1

[1616] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1617] Managing the daily eating habits of those requiring care is a significant burden for many family members and caregivers. It requires a wide range of tasks, including recording accurate dietary information, managing nutritional balance, and procuring necessary ingredients. There is a need for a support system that can reduce this management burden and help those requiring care maintain a healthy and balanced diet.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1619] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically placing orders with ingredient suppliers based on the ordering list, means for notifying the user of the ordering information, means for analyzing the accumulated meal data to evaluate the nutritional intake status, means for generating improvement suggestions if the nutritional balance is inappropriate, means for notifying the user of the suggestions, and means for analyzing past consumption patterns to predict future ingredient consumption, thereby enabling comprehensive management and support of the daily eating habits of the care recipient.

[1620] "Person requiring care" refers to an individual who requires ongoing care and support in their daily lives.

[1621] "Means for photographing meals" refers to a camera device or related equipment for recording meal contents as image data.

[1622] "Means for transmitting image data" refers to a communication means for transferring captured image data to a server.

[1623] "Means for analyzing image data to recognize the type and amount of ingredients" refers to algorithms or software that use image processing technology to identify the type and amount of ingredients.

[1624] "Means for recording the recognition results in a database" refers to methods and technologies for storing information such as the type and quantity of food in a database.

[1625] "Means for predicting the types and amounts of ingredients needed based on recorded dietary data" refers to algorithms and methods for analyzing past dietary data and predicting the types and amounts of ingredients needed in the future.

[1626] The "means for generating the next ingredient order list" refers to a mechanism for creating an order list based on the predicted types and amounts of ingredients needed.

[1627] The "means for automatically placing orders with ingredient suppliers" refers to a technology for automatically placing orders with ingredient suppliers using the generated order list.

[1628] "Means for notifying the user of order information" refers to methods and techniques for notifying the user of information regarding the order contents.

[1629] "Means for assessing nutritional intake status" refers to algorithms and methods for analyzing and assessing the intake status of individual nutrients from recorded dietary data.

[1630] "Means for generating improvement suggestions in the event of inadequate nutritional balance" refers to a system for detecting nutritional deficiencies and excesses and suggesting appropriate ingredients and dietary improvements.

[1631] "Means for notifying users of suggestions" refers to methods and techniques for notifying users of generated improvement suggestions.

[1632] "Means for predicting future food consumption by analyzing past consumption patterns" refers to algorithms and methods for analyzing past food consumption data and predicting future food consumption.

[1633] This invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management. The main components of this system are a device that takes photos of the food, a server that transmits and analyzes the image data, a database that records the recognition results and prediction results, and a means of communication with the user.

[1634] 1. Record and recognize food

[1635] Terminal

[1636] When a user eats, a camera device attached to the device automatically takes pictures of the meal. For example, a Raspberry Pi camera module is used. The captured image data is stored on the device along with date and time information, and then sent to a server via Wi-Fi or Bluetooth.

[1637] server

[1638] The server analyzes the received image data using generative AI models such as Google Cloud Vision API and TensorFlow to recognize the type and quantity of ingredients. The recognition results are recorded in a database such as MySQL. For detailed recognition, image analysis algorithms are run to detect and classify specific ingredients and their quantities.

[1639] For example, if a user has bread, salad, and coffee for breakfast, the device takes a photo of the breakfast and sends it along with the date and time to the server. The server then uses the Google Cloud Vision API to analyze the image and recognize specific ingredient information, such as "one piece of bread, one plate of salad, and one cup of coffee," and stores it in a database.

[1640] 2. Predicting necessary ingredients

[1641] server

[1642] The server analyzes past consumption patterns based on the accumulated meal records. Using Python and the pandas library, it predicts the ingredients needed and their quantities based on the analysis results. The next food ordering list is generated based on the predicted consumption amounts.

[1643] For example, if the server recognizes from the food record for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and create a list of the ingredients needed.

[1644] 3.Automatic ordering of ingredients

[1645] server

[1646] The server uses cloud services such as Amazon Web Services (AWS) to automatically place orders with partner food suppliers based on the generated order list. The order details include the ingredient name, quantity, and delivery address. The order details are notified to the user's smartphone or computer. Notifications are sent using Google Firebase.

[1647] User

[1648] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1649] For example, the server generates an ingredient list and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from an ingredient supplier. The order details are sent to the user's smartphone, and the user can check the details and make adjustments if necessary.

[1650] 4. Nutritional balance management

[1651] server

[1652] The server analyzes the nutritional intake status of the care recipient based on the recorded dietary data. It uses Python's Numpy and Scipy libraries to evaluate the nutrients in each ingredient and calculate the intake amount. Based on the results of the nutritional balance evaluation, if there is a nutrient deficiency or excess, a suggestion is made to add appropriate ingredients. This suggestion is then sent to the user's smartphone.

[1653] User

[1654] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1655] For example, if the past week's records show that a care recipient has an iron deficiency, the server will send a notification to the user's smartphone with advice such as "We recommend adding iron-rich foods such as liver and spinach." The user can then improve their diet based on this advice.

[1656] Prompt Sentence Examples

[1657] By inputting the following prompt sentences into the generative AI model, meal recognition and nutritional analysis functions can be realized.

[1658] "Please tell me the algorithm to analyze images of meals taken by care recipients and recognize the types and portions of ingredients."

[1659] "How can I predict what ingredients and amounts I'll need next week based on my diet data from the past week?"

[1660] "Please explain how to analyze recorded dietary data and generate suggestions for improving nutritional balance."

[1661] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1662] Processing Steps

[1663] Step 1: Photograph your food

[1664] Terminal

[1665] A camera device attached to the device automatically takes pictures of the user's meal. Specifically, the image data is acquired using a Raspberry Pi camera module. The input is the image captured by the camera, and the output is the image data that is stored on the device.

[1666] Step 2: Sending image data

[1667] Terminal

[1668] The device sends the captured image data to the server via Wi-Fi or Bluetooth. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, an HTTP POST request is sent using the Python requests library.

[1669] Step 3: Image analysis

[1670] server

[1671] The server analyzes the image data using image analysis algorithms such as Google Cloud Vision API and TensorFlow. The input is the image data sent to the server, and the output is the analysis results. Specifically, the processing recognizes the types and quantities of ingredients contained in the image.

[1672] Step 4: Save the recognition results to a database

[1673] server

[1674] The server records the results of image analysis in a database such as MySQL. The input is the recognition results obtained by image analysis, and the output is a meal record stored in the database. Specifically, the recognition results are inserted into the database using an SQL query.

[1675] Step 5: Analyze consumption patterns

[1676] server

[1677] The server analyzes past consumption patterns based on the accumulated food record data. The input is the food records stored in the database, and the output is the analysis results of consumption patterns. Specifically, the past data is aggregated and analyzed using the Python pandas library.

[1678] Step 6: Generate an ingredient ordering list

[1679] server

[1680] The server generates the next ingredient order list based on the analysis results. The input is the result of consumption pattern analysis, and the output is an order list. Specifically, it lists the types and quantities of ingredients needed.

[1681] Step 7: Automated ordering of ingredients

[1682] server

[1683] The server automatically places orders with retailers based on the generated order list. The input is the order list, and the output is the execution result of the order. Specifically, the order procedure is carried out using AWS Lambda and API Gateway.

[1684] Step 8: Notification of order details

[1685] server

[1686] The server notifies the user of the order details. The input is the execution result of the order, and the output is the notification to the user. Specifically, the notification is sent using Google Firebase.

[1687] Step 9: Analyze nutritional status

[1688] server

[1689] The server analyzes the user's nutritional intake status based on the recorded dietary data. The input is the dietary records stored in the database, and the output is the evaluation results of the nutritional intake status. Specific processing involves using Python's Numpy and Scipy libraries to evaluate the nutrients in each food ingredient and calculate the intake amount.

[1690] Step 10: Suggestions for improving nutritional balance

[1691] server

[1692] The server evaluates nutritional balance and generates suggestions for adding appropriate ingredients to address nutrient deficiencies or excesses. The input is the assessment result of nutritional intake status, and the output is improvement suggestions. Specific operations include notifying the user of the generated suggestions.

[1693] (Application example 1)

[1694] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1695] It is important to efficiently manage the daily dietary content of those requiring care and maintain an appropriate nutritional balance. However, current systems only partially address the dietary management of those requiring care and the automatic ordering of ingredients, making it difficult to comprehensively manage nutritional balance and resolve food shortages. Furthermore, no rational solutions are provided, including inventory management and automatic ordering at physical stores.

[1696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1697] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to the server, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a database, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from ingredient suppliers based on the ordering list, means for supporting ingredient management and inventory replenishment for the care recipient at a physical store, means for notifying the user of ordering information, means for analyzing the nutritional intake status of the care recipient, means for evaluating nutritional balance and generating improvement suggestions if inadequate, and means for analyzing past consumption patterns based on the care recipient's meal records to predict future consumption. This enables efficient dietary management of the care recipient, maintaining nutritional balance, and automatic replenishment of necessary ingredients.

[1698] "Care recipient" refers to an individual who requires assistance from others in daily living.

[1699] "Means for photographing meals" refers to a camera or image acquisition device for photographing the meals consumed by the person requiring care.

[1700] "Means for transmitting image data to a server" refers to means for sending image data of photographed food to a server via the Internet or other communication means.

[1701] "Means for analyzing image data" refers to algorithms or software that analyze captured image data and recognize the type and quantity of ingredients.

[1702] "Means for recording in a database" refers to a database management system that stores the analyzed data and allows it to be referenced later.

[1703] "Means for predicting the types and amounts of ingredients needed" refers to a system for predicting the types and amounts of ingredients needed in the future based on recorded past dietary data.

[1704] "Means for generating an ingredient order list" refers to software or algorithms that automatically generate the next order list based on the predicted ingredient needs and quantities.

[1705] The "means for automatically placing orders with food ingredient suppliers" refers to a means for automatically placing orders for ingredients with affiliated food ingredient suppliers based on the generated order list.

[1706] "A means to support food ingredient management and inventory replenishment for people requiring care in physical stores" refers to a system that uses the meal records and consumption patterns of people requiring care to streamline food ingredient inventory management and replenishment in physical stores.

[1707] The "means for notifying the user of order information" refers to a means for notifying the user of the generated order list and information related to the order contents.

[1708] "Means for analyzing nutritional intake status" refers to software or algorithms for analyzing the types and amounts of nutrients being ingested based on the dietary data of the person requiring care.

[1709] "Means for assessing nutritional balance and proposing improvements if inappropriate" refers to a system for assessing nutritional balance based on nutritional intake status and generating specific improvement proposals to supplement missing nutrients.

[1710] "Means for analyzing past consumption patterns to predict future consumption" refers to algorithms and software that analyze consumption patterns from recorded dietary data and predict future consumption.

[1711] The present invention provides a system for efficiently managing the daily dietary content of a care-requiring person and automating food ingredient management and inventory replenishment in a physical store while maintaining appropriate nutritional balance. Specific embodiments for implementing the present invention are described below.

[1712] 1. System Configuration

[1713] The system mainly consists of the following components:

[1714] A camera device for taking photos of meals for care recipients

[1715] A communication terminal for transmitting image data to a server

[1716] Software that analyzes image data to recognize the type and quantity of ingredients

[1717] A database management system for recording analysis results in a database

[1718] An algorithm that predicts the ingredients and amounts needed based on recorded meal data

[1719] Software for generating next food ordering list

[1720] A system to support food management and inventory replenishment in brick-and-mortar stores

[1721] Communication methods to notify users of ordering information and offers

[1722] Nutritional status analysis software used to analyze results

[1723] Software for assessing nutritional balance and generating improvement suggestions

[1724] Algorithms for analyzing past consumption patterns and predicting future consumption

[1725] 2. Program Overview

[1726] Hardware and Software Usage:

[1727] Hardware: Smartphones, camera devices

[1728] Software: Python, OpenCV, Requests, Web server, Database (e.g. MySQL)

[1729] Process flow:

[1730] 1. Image capture:

[1731] The terminal uses a camera device to take an image of the meal when the person requiring care eats the meal.

[1732] 2. Sending images:

[1733] The captured image data is sent via the device to a server via the Internet, allowing the server to receive information about the meal in real time.

[1734] 3. Image analysis and recognition:

[1735] The server analyzes the image data using Python and OpenCV to identify the type and quantity of ingredients, and the analyzed data is stored in a database.

[1736] 4. Forecast and order ingredients:

[1737] The server runs an algorithm (e.g., a machine learning model) that predicts the type and amount of ingredients needed next based on the recorded meal data. Based on the results, it generates a list of ingredients to order next and notifies the physical store.

[1738] 5. Inventory management in physical stores:

[1739] The physical store manages inventory based on the received order list and replenishes supplies to ensure necessary ingredients are available.

[1740] 6. User Notification and Nutrition Management:

[1741] The server notifies the user of the order information and the analysis results of the nutritional intake status. If the nutritional balance is inappropriate, the server generates improvement suggestions and sends them to the user.

[1742] 3. Specific Examples

[1743] If the server recognizes that bread, salad, and coffee are the breakfast items for a care recipient, it analyzes the consumption patterns of these items based on past data and automatically places an order with a physical store to replenish any missing ingredients.

[1744] Example prompt for a generative AI model:

[1745] "Please analyze the following food image and tell me the types and amounts of ingredients contained in it."

[1746] This system will improve the accuracy of dietary management for those requiring care and also streamline inventory management of food ingredients in physical stores.

[1747] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1748] Step 1:

[1749] The device uses a camera device to capture images of the meal while the care recipient is eating, and the captured images are given date and time information.

[1750] Input: Meal image, date and time information

[1751] Output: Image file of the photographed meal, date and time information

[1752] Step 2:

[1753] The terminal transmits the captured image data to a server using a communication means, and the image data is uploaded to the server via the Internet.

[1754] Input: Meal image file, date and time information

[1755] Output: Image file of the meal sent to the server, date and time information

[1756] Step 3:

[1757] The server analyzes the received image data using Python and OpenCV. This analysis identifies the type and quantity of ingredients. The analysis results include the identified type of ingredients, quantity, and date and time information.

[1758] Input: Image file of the meal sent to the server, date and time information

[1759] Data processing: Applying image analysis algorithms to identify ingredient types and quantities

[1760] Output: Identified ingredients, quantity, date and time

[1761] Step 4:

[1762] The server records the analysis results in a database, which stores the date and time of the meal, the type of ingredients identified, and the portion size.

[1763] Input: Identified ingredient type, quantity, date and time information

[1764] Data processing: Convert to database format

[1765] Output: Meal data stored in a database

[1766] Step 5:

[1767] The server predicts the ingredients needed and their quantities based on dietary data stored in a database, and the prediction is made by analyzing past consumption patterns.

[1768] Input: Food data stored in a database

[1769] Data Computing: Predicting future consumption using machine learning models

[1770] Output: Predicted ingredients and quantities needed

[1771] Step 6:

[1772] The server generates an order list for the next meal based on the predictions, which includes the types and quantities of ingredients needed.

[1773] Input: Predicted ingredients and quantities needed

[1774] Data processing: Convert to order list format

[1775] Output: Order list

[1776] Step 7:

[1777] The server automatically places orders with partner food suppliers based on the generated order list. The order details include the name, quantity, and delivery address of the required ingredients.

[1778] Input: Purchase Order List

[1779] Data calculation: Apply automatic ordering algorithm

[1780] Output: Order notification to ingredient supplier

[1781] Step 8:

[1782] The server notifies the user of the order information, who can then check the order details via a device such as a smartphone and adjust the order quantity if necessary.

[1783] Input: Generated purchase order list, purchase order notification

[1784] Output: User notification, confirmation result

[1785] Step 9:

[1786] The server analyzes the nutritional intake status of the care recipient, evaluates the nutrients ingested based on the dietary data, and generates appropriate improvement suggestions if there are any deficiencies.

[1787] Input: Food data stored in a database

[1788] Data calculation: Applying nutrient analysis algorithms

[1789] Output: Nutritional status report, improvement suggestions

[1790] Step 10:

[1791] The server then notifies the user of the nutritional improvement suggestions it has generated, allowing the user to adjust their diet based on this information.

[1792] Input: Nutritional status report, improvement suggestions

[1793] Output: User notification, adjustment results

[1794] As a concrete example, if a care recipient has bread, salad, and coffee for breakfast, the device takes a photo of it and sends it to the server. The server analyzes the image, recognizes the types and amounts of ingredients, and records them in a database. It then predicts the amount of ingredients needed based on past meal data, generates an order list for the physical store, places the order, and notifies the user. The server also provides information based on prompts such as, "Analyze the following meal image and tell me the types and amounts of ingredients it contains."

[1795] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1796] The present invention is a system that recognizes the daily dietary contents of a care-requiring person, automatically orders the necessary ingredients, and supports nutritional management, and also combines this with an emotion engine that recognizes the user's emotions. The main components of this system are as follows:

[1797] 1. Record and recognize food

[1798] Terminal

[1799] When the care recipient eats, the robot or camera device takes a photo of the meal, and the captured image is saved on the device along with the date and time.

[1800] The terminal transmits the stored image data to the server.

[1801] server

[1802] The server analyzes the received image data and uses image analysis algorithms to identify the type and quantity of ingredients.

[1803] The server records the results of the recognition in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1804] Specific examples

[1805] When a care recipient has bread, salad, and coffee for breakfast, the device takes a photo and sends it along with the date and time information to the server. The server analyzes the image, recognizes specific ingredient information, such as "one piece of bread, one plate of salad, one cup of coffee," and stores it in a database.

[1806] 2. Predicting necessary ingredients

[1807] server

[1808] Based on the accumulated food records, past consumption patterns are analyzed, and based on the analysis results, the ingredients and amounts needed are predicted.

[1809] Generate the next ingredient order list. The order list contains the types of ingredients and the required quantities.

[1810] Specific examples

[1811] If the server recognizes from the food log for the past week that seven loaves of bread, two bags of salad vegetables, and seven cups of coffee have been consumed, it will predict the same consumption amounts for the next week and list the ingredients needed.

[1812] 3.Automatic ordering of ingredients

[1813] server

[1814] Orders are automatically placed with partner food suppliers based on a predicted order list, including the name of the food, the quantity, and the delivery address.

[1815] The order contents and detailed information are notified to the user, who can then check the order contents via their smartphone or computer and make adjustments as necessary.

[1816] User

[1817] The user checks the order details and adjusts the order quantity as necessary, and this adjustment information is sent back to the server.

[1818] Specific examples

[1819] The server generates a list of ingredients and orders "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" from the retailer. The order details are sent to the family's smartphones, where they can check the details and make adjustments if necessary.

[1820] 4. Nutritional balance management

[1821] server

[1822] Based on the recorded dietary data, the nutritional intake status of the care recipient is analyzed, evaluating the nutrients in each food ingredient and calculating the intake amount.

[1823] Based on the nutritional balance assessment results, if there is a nutrient deficiency or excess, a suggestion to add appropriate ingredients is generated.

[1824] The generated nutritional balance improvement proposal is notified to the user.

[1825] User

[1826] Users can review their nutritional status report and suggestions for improvement, and make improvements or adjustments to their diet.

[1827] Specific examples

[1828] If the past week's records show that the care recipient has an iron deficiency, the server generates advice such as "We recommend adding iron-rich foods such as liver and spinach," and sends it to the family's smartphone. Family members can then improve their diet based on this advice.

[1829] 5. Combination of Emotion Engines

[1830] server

[1831] The server incorporates an emotion engine for recognizing the user's emotions, which analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[1832] It runs algorithms that tailor orders and nutritional suggestions based on perceived emotions, for example, suggesting foods with a relaxing effect if the user is feeling stressed.

[1833] User

[1834] The notification method is also adjusted depending on the user's emotional state: if the user is feeling anxious, for example, the notification will be delivered in a calmer manner.

[1835] Specific examples

[1836] If the user is feeling stressed while confirming their order, the emotion engine will recognize their emotions and suggest adding relaxing herbal tea to the order. The notification will also be delivered in a calm tone, helping to reduce the user's stress.

[1837] This system comprehensively manages the diet of those requiring care, maintaining an appropriate nutritional balance while taking into account the user's emotional state, significantly reducing the burden on family members and caregivers.

[1838] The processing flow will be explained below.

[1839] Step 1:

[1840] Device: When the care recipient eats, the robot or camera device takes a photo of the meal. The captured image data and date and time information are saved on the device.

[1841] Step 2:

[1842] Device: Sends the saved image data and date and time information to the server.

[1843] Step 3:

[1844] Server: The received image data is passed through an image analysis algorithm. A convolutional neural network (CNN) is used to recognize the type and quantity of ingredients from the image data.

[1845] Step 4:

[1846] Server: Records the analysis results in a database, including the date and time of the meal, the type of ingredients, and the portion size.

[1847] Step 5:

[1848] Server: Retrieves past meal records from the database and compiles the frequency and consumption of ingredients used.

[1849] Step 6:

[1850] Server: Predicts the amount and ingredients needed based on the aggregated results. Analyzes past consumption patterns and creates a list of the ingredients and quantities needed next time.

[1851] Step 7:

[1852] Server: Generates the next ingredient order list based on the forecast. The order list includes the ingredient names and required quantities.

[1853] Step 8:

[1854] Server: Automatically places orders with partner food suppliers based on the order list. Sends the order details to the supplier as a request.

[1855] Step 9:

[1856] Server: Sends order details and notification to family members or caregivers. Notifications include order details and a confirmation link to their smartphone or computer.

[1857] Step 10:

[1858] User: Family members or caregivers can check the order details via their smartphone or computer and make any necessary adjustments. The adjusted details are then sent back to the server.

[1859] Step 11:

[1860] Server: Analyzes the nutritional intake status of the care recipient based on the recorded dietary data, evaluates the nutrients of each ingredient, and calculates the intake amount.

[1861] Step 12:

[1862] Server: Based on the results of the nutritional balance assessment, if there is a nutrient deficiency or excess, it generates improvement suggestions and notifies the family or caregiver of the suggestions.

[1863] Step 13:

[1864] Users: Family members and caregivers can review the reports and improvement suggestions provided to improve their diet and adjust nutritional balance.

[1865] Step 14:

[1866] Device: Uses a camera and microphone to capture your facial expressions and tone of voice as you review your meal and order details.

[1867] Step 15:

[1868] Device: Sends the captured data to the emotion engine, which analyzes the user's emotional state and sends the results to the server.

[1869] Step 16:

[1870] Server: Adjusts the order and nutritional suggestions based on the emotional state received from the emotion engine. For example, if the user is feeling stressed, it will suggest adding ingredients that have a relaxing effect.

[1871] Step 17:

[1872] Server: Notifies family members or caregivers with adjusted orders and nutritional suggestions. Notifications are also tailored to the user's emotional state. For example, if the user is feeling anxious, the notification will be delivered in a calming manner.

[1873] Step 18:

[1874] User: Family or caregiver reviews the order and nutrition suggestions provided and makes adjustments as needed.

[1875] Through these specific processing steps, the diet of the person requiring care can be efficiently managed, maintaining an appropriate nutritional balance while taking into consideration the user's emotional state, significantly reducing the burden on family members and caregivers.

[1876] Example 2

[1877] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1878] While daily dietary management and nutritional balance for those requiring care are extremely important, they often place a significant burden on family members and caregivers. Furthermore, because it is necessary to consider the emotional state of the care recipient, a system that can do this efficiently is needed. While existing systems partially cover automatic ordering of ingredients and nutritional balance management, they lack functionality that takes the user's emotional state into account, and are therefore still insufficient. Therefore, there is a need to provide a system that can solve these issues and more effectively manage the diet of those requiring care while also taking the user's emotional state into consideration.

[1879] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1880] In this invention, the server includes means for photographing the meals of the care recipient, means for transmitting the photographed image data to an information processing device, means for analyzing the image data to recognize the types and amounts of ingredients, means for recording the recognition results in a storage device, means for predicting the ingredients and their amounts needed based on the recorded meal data, means for generating a next ingredient ordering list based on the prediction, means for automatically ordering from suppliers based on the ordering list, means for notifying the user of the ordering information, means for recognizing the user's emotional state, and means for adjusting the order content and notification method based on the user's emotional state. This not only enables efficient dietary management and maintenance of nutritional balance for the care recipient, but also enables appropriate responses that take the user's emotional state into consideration.

[1881] "Persons requiring care" refers to people who have difficulty managing some or all of their daily lives on their own and require care and support.

[1882] "Meals" refers to the food and drink consumed daily by the person requiring care.

[1883] "Means for taking photographs" refers to a device for recording images of the meal, such as a camera device or a robot.

[1884] "Image data" refers to image information of the photographed meal.

[1885] "Information processing device" refers to a device for transmitting, receiving, and analyzing data, such as a server or computer system.

[1886] "Means for analysis" refers to the algorithms and software used to analyze image data and recognize the type and quantity of ingredients.

[1887] "Ingredients" refer to the individual food elements in a meal.

[1888] "Type" refers to the classification of multiple ingredients that are recognized.

[1889] "Quantity" refers to the quantity or volume of each ingredient.

[1890] "Recognition results" refers to information on the type and quantity of ingredients obtained from the analyzed image data.

[1891] "Storage device" refers to a device that stores information, such as a database or recording medium.

[1892] "Necessary ingredients" refers to foods that the care recipient is expected to consume in future meals.

[1893] "Order List" refers to a list of predicted ingredients and their quantities.

[1894] "Supplier" refers to a company that provides ingredients based on an order list.

[1895] "Means for automated ordering" refers to a system or process for automatically placing orders with suppliers based on an order list.

[1896] "Order information" refers to the type and quantity of ingredients ordered, delivery address information, etc.

[1897] "Means of notification" refers to the methods and techniques used to convey information to users.

[1898] "Emotional state" refers to the current mental and emotional state of the user or care recipient.

[1899] "Means of recognition" refers to technologies and systems for detecting and analyzing emotional states.

[1900] "Adjusting means" refers to a process or device for changing order content or notification methods based on emotional state.

[1901] This invention is a system that recognizes the daily dietary contents of a care recipient, automatically orders the necessary ingredients, and supports nutritional management, and also combines it with an emotion engine that recognizes the user's emotions. The main components of this system are three entities: a terminal, a server, and a user.

[1902] First, the device has a means to take pictures of the meal when the care recipient eats. Specifically, the pictures are taken using a robot or a camera device. The captured image data is saved on the device along with date and time information, and the saved data is sent to the server after undergoing preprocessing. This preprocessing includes resizing and normalizing the image.

[1903] The server then analyzes the received image data to identify the type and quantity of ingredients. The image analysis algorithm used here includes, for example, an image classification model using TensorFlow. The ingredient information obtained as a result of the recognition is stored in a database, which is a storage device. The stored information includes the type of ingredient, the quantity, and the date and time of the meal.

[1904] The server also has a way to predict what ingredients will be needed and in what quantities, based on the accumulated meal records. For this analysis, it uses the machine learning library Scikit-learn, for example, to predict future consumption based on past consumption patterns. It then generates the next ingredient ordering list based on the prediction results.

[1905] The server also includes a means for automatically sending the generated order list to suppliers. For example, using the Twilio API, an order request is sent along with information on the ingredients, quantity, and delivery address. The details of the order are notified to the user, who can check them via an application on their smartphone or computer. The user can then check the order and make adjustments as necessary. The adjusted information is then sent back to the server.

[1906] The server also has analytical tools for managing the nutritional balance of care recipients. Based on the recorded dietary data, it evaluates the nutrients in each ingredient using, for example, Pandas, and calculates the intake amount. Based on the results of the nutritional balance evaluation, it also generates suggestions for adding appropriate ingredients. These suggestions are also notified to the user.

[1907] The system also incorporates an emotion engine that recognizes the user's emotional state. The server runs emotion recognition algorithms, using Microsoft Azure's Emotion API to analyze the user's facial expressions and tone of voice, for example. It also includes a means to adjust order content and notification methods based on the recognized emotional state. For example, if it determines that the user is feeling stressed, an algorithm will be run to suggest foods with a relaxing effect.

[1908] Specific examples

[1909] As a concrete example, suppose a user takes a photo of a care recipient's breakfast. The image is taken by the device and sent to the server along with the date and time information. The server analyzes the image, recognizes it as "one piece of bread, one plate of salad, and one cup of coffee," and saves it in a database. Based on the past week's meal records, the server predicts the next meal's necessary ingredients: seven pieces of bread, two bags of salad vegetables, and one bag of coffee, and places an order with the supplier. The order details are notified to the user's smartphone, and the user can review the order and make adjustments if necessary. If the server determines that the care recipient is iron deficient, it will generate advice such as "it is recommended to add liver or spinach" and notify the user. In addition, if the server recognizes that the user is feeling stressed while confirming the order details, it will add a suggestion of relaxing herbal tea.

[1910] Prompt Sentence Examples

[1911] "Please describe the process flow for a system that takes photos of the meals of a care recipient and automatically orders ingredients for the next meal and manages nutrition. Please also explain how the system can be combined with an emotion engine to take the user's emotional state into account."

[1912] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1913] Step 1: Photograph your meal and send the data

[1914] Device:

[1915] When the care recipient begins to eat, the camera device takes an image of the meal.

[1916] Input: Image from camera device

[1917] Data processing: resizing and normalizing images

[1918] Output: Preprocessed image

[1919] Specific operation: When the care recipient eats breakfast, the camera automatically takes a photo, and the image is saved on the device along with the date and time information. The pre-processed image is then sent to the server.

[1920] Step 2: Recognizing ingredients through image analysis

[1921] server:

[1922] The received image data is analyzed to recognize the type and quantity of ingredients.

[1923] Input: Preprocessed image data

[1924] Data computation: Analysis using image classification models with TensorFlow

[1925] Output: Data on the type and quantity of recognized ingredients

[1926] Specific operation: The server analyzes the received image, recognizes specific ingredient information such as "1 piece of bread, 1 plate of salad, 1 cup of coffee," and stores it in a database.

[1927] Step 3: Save your meal data

[1928] server:

[1929] The analyzed results are saved in a database.

[1930] Input: Recognized ingredient type and quantity

[1931] Data processing: Converting data into a format suitable for database format

[1932] Output: Success / failure of saving to database

[1933] Specific operation: The recognized ingredient information is saved in a database as "meal date and time, type of ingredient, and quantity."

[1934] Step 4: Predict the ingredients you need

[1935] server:

[1936] Based on accumulated meal data, it predicts the ingredients and amounts needed next time.

[1937] Input: Food data stored in the database

[1938] Data Computing: Analyzing historical consumption patterns with machine learning models using Scikit-learn

[1939] Output: Next food order list

[1940] Specific operation: The server analyzes the food data from the past week and predicts that "next week you will need 7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee."

[1941] Step 5: Automated ordering of ingredients

[1942] server:

[1943] Based on the predicted order list, orders are automatically placed with suppliers.

[1944] Input: Next food order list

[1945] Data processing: Convert the order list into an API request format

[1946] Output: Success / failure of sending purchase order request to supplier

[1947] Specific operation: The server places an order for "7 loaves of bread, 2 bags of salad vegetables, and 1 bag of coffee" with the supplier's API, and the order details are notified to the user.

[1948] Step 6: User confirms and adjusts order

[1949] User:

[1950] Review orders and make adjustments as necessary.

[1951] Input: Order Notification

[1952] Data processing: Sending adjusted order information

[1953] Output: Success / failure of sending adjusted order information to the server

[1954] Specific operation: The user checks the order details on their smartphone or computer, adjusts the order quantity if necessary, and sends the adjustment information back to the server.

[1955] Step 7: Nutritional Balance

[1956] server:

[1957] Based on the recorded dietary data, nutritional intake status is analyzed and nutritional balance is evaluated.

[1958] Input: Food data stored in the database

[1959] Data calculation: Nutrient evaluation and calculation using the Pandas library

[1960] Output: Nutritional balance assessment results and improvement suggestions

[1961] Specific operation: If the user is found to be iron deficient based on the records from the past week, advice such as "It is recommended to add liver or spinach" is generated and notified to the user.

[1962] Step 8: Recognize your emotional state

[1963] server:

[1964] Analyze the user's emotional state and adjust notifications and order content.

[1965] Input: User facial and voice data

[1966] Data Computing: Sentiment Analysis using Microsoft Azure's Emotion API

[1967] Output: Perceived emotional state

[1968] What it does: If the user is feeling stressed while reviewing their order, the server will recognize this and add a suggestion for a relaxing herbal tea.

[1969] These steps will enable the system to manage the diet of those in need of care and maintain nutritional balance, while also taking into account the emotional state of the user.

[1970] (Application example 2)

[1971] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1972] In an aging society, managing the diet of those requiring care is a significant burden for families and caregivers. Maintaining nutritional balance and properly ordering ingredients is time-consuming, making consistent management difficult. Furthermore, there is a need to understand the emotional state of those requiring care and adjust ingredients and notifications accordingly, but currently this is often done manually and is inefficient.

[1973] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of the nursing care recipient's mealtime behavior, a means for transmitting the captured image information to the information management device, a means for analyzing the image information to recognize the type and quantity of food, a means for saving the recognition results in a recording device, a means for predicting the food items and their quantities based on the recorded lunch data, a means for generating a next food order list based on the prediction, a means for automatically placing orders with a food provider based on the order list, a means for notifying the user of the order information, a means for recognizing the user's emotional state, and a means for adjusting the order content and notification method based on the recognized emotion. This automates the nursing care recipient's dietary management, enabling proper maintenance of nutritional balance and efficient ordering of ingredients. Furthermore, the server automatically suggests ingredients and adjusts the notification method based on the user's emotional state, reducing the burden of caregiving.

[1974] A "person requiring care" is an elderly person or a disabled person who requires assistance from others in daily life.

[1975] "Food-feeding behavior" refers to the act and state of consuming food.

[1976] "Means for taking photographs" refers to a method of recording the meal as images or videos using a device such as a camera.

[1977] "Image information" refers to data of captured images or videos.

[1978] An "information management device" refers to a server or cloud service that receives and stores captured image information.

[1979] The "means of analysis" refers to a method of identifying the type and quantity of ingredients from image information using artificial intelligence or image recognition algorithms.

[1980] A "recording device" refers to a database or storage system for saving analysis results.

[1981] "Meal data" refers to recorded information regarding the dietary content of a care recipient.

[1982] "Prediction methods" are methods for calculating the types and quantities of food that will be needed in the future based on past school lunch data.

[1983] "Order List" means a list of the types and quantities of food items that are expected to be needed.

[1984] "Food provider" means a supplier that provides food needs based on a forecast.

[1985] "Means for automatically ordering" means the method by which the system automatically orders the required food from the food provider.

[1986] "Means of notification" refers to the method used to notify users of order information and system suggestions. This can primarily be electronic means such as smartphones or email.

[1987] "Means for recognizing emotional states" refers to algorithms or devices that analyze facial expressions and voice data of the care recipient and their caregiver to identify emotions.

[1988] "Adjustment means" are methods for changing the content of suggestions or notification methods based on the perceived emotional state.

[1989] This system automates the dietary management of people requiring care, maintaining nutritional balance and efficiently ordering ingredients. The system also recognizes the user's emotional state and adjusts ingredient suggestions and notifications accordingly.

[1990] 1. Required Hardware and Software

[1991] Hardware:

[1992] Smartphone with camera: Used to take photos of the mealtime behavior of the care recipient

[1993] Server: Analyzes and stores data, and manages orders

[1994] software:

[1995] OpenCV: A library for image capture and processing

[1996] EmotionRecognition: A library for emotion analysis using facial and voice data

[1997] Flask / Django: Frameworks for implementing server-side APIs

[1998] 2. Detailed System Configuration

[1999] a. Photographing feeding behavior and sending data

[2000] The smartphone camera is used to take pictures of the person receiving care eating. This image information is sent from the smartphone to a server. A dedicated application is installed on the smartphone, automating the process of taking pictures and sending the images.

[2001] b. Image analysis and food ingredient recognition

[2002] The server analyzes the received image data and identifies the type and quantity of ingredients. This analysis uses an image recognition algorithm incorporating a deep learning model. The results are stored in a database and later used for evaluating nutritional balance and ordering food.

[2003] c. Forecasting food needs and generating an ordering list

[2004] The server uses past meal data to calculate the type and quantity of food needed for the next meal. Based on the prediction, a next order list is generated. This order list is automatically sent to the food provider.

[2005] d. Evaluation of nutritional balance and suggestions for improvement

[2006] The server analyzes the nutritional intake status of the care recipient based on the recorded meal...

Claims

1. A means of photographing meals for care recipients, means for transmitting the captured image data to a server; A means for analyzing image data to recognize the type and amount of ingredients; means for recording the recognition results in a database; A method to predict the ingredients and their amounts needed based on recorded dietary data, A means for generating a next food ordering list based on the prediction; a means for automatically placing orders with food suppliers based on the order list; A system including a means for notifying a user of order information.

2. A means for analyzing the nutritional intake status of a care recipient; A means for evaluating nutritional balance and generating improvement suggestions if inadequate; 10. The system of claim 1, further comprising means for notifying the user of the suggestion.

3. 2. The system according to claim 1, further comprising means for analyzing past consumption patterns based on the dietary records of the person requiring care and predicting future consumption amounts.

Images