Eating status estimation apparatus, estimation method, and program

Abstract

To estimate an eating status of a meal with higher accuracy by taking into consideration changes and variations in how food is served on tableware and how the food is eaten.SOLUTION: An eating status estimation apparatus acquires a first food image captured before eating and a second food image captured after eating for food to be eaten, extracts a first edible portion before eating from the first food image, and extracts a second edible portion after eating from the second food image. The apparatus estimates an eating status of the food on the basis of a change in an area between the first edible portion and the second edible portion, estimates densities of the food in the first edible portion and the second edible portion, and corrects an estimation result of the eating status of the food on the basis of the densities of the food.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] One aspect of the present invention relates to an eating status estimation device, an estimation method, and a program used to estimate the eating status of hospitalized patients and residents in, for example, hospitals and nursing homes. [Background technology]

[0002] In medical and welfare settings, understanding the nutritional status of patients and care recipients is extremely important for managing their recovery from injury or illness and their health. For this reason, hospitals and nursing homes check the dietary intake of patients and residents.

[0003] Common methods for checking meal intake include having medical professionals or nutritionists directly ask patients or residents about their meal intake, or visually comparing the state of their dishes before and after eating and recording the results. However, this method is prone to inconsistency in staff judgment of meal intake, and requires a lot of time and effort to conduct the survey, placing a heavy burden on staff and making it inefficient.

[0004] Therefore, a method has been proposed to estimate the eating status of a meal by taking images of food served on a plate before and after eating, and then processing each of the images to calculate, for example, the area ratio of the edible parts (see, for example, Patent Documents 1 or 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-86313 [Patent Document 2] International Publication No. 2021 / 085369 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the techniques described in Patent Documents 1 and 2 basically estimate food intake from the ratio of the area of ​​food on the dish before and after eating. Therefore, it is difficult to accurately estimate food intake depending on how food is served on the dish and how the food is eaten. For example, when food is served in a large amount vertically, if the area of ​​the edible portion changes little as the person continues to eat but the height decreases, the error between the estimated food intake and the actual food intake becomes large.

[0007] Patent Document 1 also describes a technology for converting food area ratios into volume ratios. This technology converts area into volume using fixed conversion information prepared in advance for each food ingredient and type of dish. However, the way food is presented on dishes and the way people eat vary, and even for the same person, it is not always consistent. For this reason, it is difficult to estimate food intake taking into account changes and variations in food presentation and eating habits.

[0008] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that can estimate eating habits with higher accuracy, taking into account changes and variations in how food is served on dishes and how people eat. [Means for solving the problem]

[0009] To solve the above problems, one aspect of an eating status estimation device or method according to the present invention acquires a first food image taken before eating and a second food image taken after eating for a food to be eaten, extracts a first edible part before eating from the first food image, and extracts a second edible part after eating from the second food image, estimates the eating status of the food based on the change in area between the first edible part and the second edible part, estimates the density of the food in the first edible part and the second edible part, and corrects the estimation result of the eating status of the food based on the density of the food.

[0010] According to one aspect of the present invention, for example, in cases where the area of ​​the edible part does not change significantly between before and after eating but the height of the food does change, the estimated result of the eating status is corrected based on the density of the food, making it possible to estimate the eating status with higher accuracy than when estimating it based on area alone. [Effects of the Invention]

[0011] That is, according to one aspect of the present invention, it is possible to provide a technology that can estimate the eating status of a meal with higher accuracy, taking into account changes and variations in how food is served and how it is eaten. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram used to explain the outline of an eating status estimating device according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of an eating status estimating device according to an embodiment of the present invention. [Figure 3] FIG. 3 is a block diagram showing an example of the software configuration of the eating status estimating device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing an example of the procedure and content of the eating status estimation process executed by the control unit of the eating status estimation device shown in FIG. [Figure 5] FIG. 5 is a diagram for explaining an example of the process of estimating edible area before and after eating, which is part of the eating state estimation process shown in FIG. [Figure 6] FIG. 6 is a diagram for explaining an example of a case in which the results of estimating the density of edible parts before and after eating are required in the eating state estimating process shown in FIG. [Figure 7] FIG. 7 is a diagram for explaining an example of the remaining amount classification process in the eating state estimation process shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] [One embodiment] (overview) FIG. 1 is a diagram for explaining an outline of an eating status estimating device according to an embodiment of the present invention.

[0015] The eating status estimation device according to one embodiment of the present invention comprises an image processing unit 100, an edible portion area estimation unit 200, a food density estimation unit 300, a remaining food amount classification unit 400, and an eating status correction unit 500.

[0016] The image processing unit 100 acquires, for example, an image of food served on tableware taken before eating and an image of the food taken after eating, using a camera (not shown).The image processing unit 100 then extracts images representing edible parts from each of the acquired pre-meal and post-meal food images.

[0017] The edible area estimation unit 200 calculates the area difference or area ratio of the edible parts before and after eating based on the images representing each edible part before and after eating extracted by the image processing unit 100, and estimates, for example, the amount of food eaten or the amount remaining after eating as information representing the food eating situation based on the calculated area difference or area ratio.

[0018] The food density estimation unit 300 estimates the density of food in the edible parts based on the images representing the edible parts before and after eating extracted by the image processing unit 100.

[0019] The food remaining amount classification unit 400 classifies the state of food remaining amount into either "completely eaten," "not eaten," or "partially eaten" based on each image representing each edible part before and after eating extracted by the image processing unit 100.

[0020] The eating status correction unit 500 corrects the food eating status estimation result obtained by the edible portion area estimation unit 200 based on the food density estimated by the food density estimation unit 300 and the food remaining amount classification result obtained by the food remaining amount classification unit 400, and outputs the corrected food eating status estimation result.

[0021] With the eating status estimation device configured as described above, the estimation result of the food eating status obtained based on the area of ​​the edible parts before and after eating is corrected based on the density of the edible parts. Therefore, for example, in a case where the area of ​​the edible parts before and after eating does not change much but the height of the food changes, the estimation result of the eating status is corrected based on the density of the food, making it possible to estimate the eating status with higher accuracy than when estimating based on area alone.

[0022] Furthermore, the eating status estimation result obtained based on area is corrected based on the classification result of the remaining food amount. Therefore, for example, even if there is no food left on the dish after a meal but sauce remains and this sauce is detected as the area of ​​the edible portion, the remaining food amount is classified as "completely eaten," so the eating status estimation result based on area is corrected based on this classification result. Therefore, it is possible to estimate the eating status with higher accuracy than when estimating based on area alone.

[0023] (Configuration example) (1) System In one embodiment of the system of the present invention, for example, a tray with tableware containing food is placed on an inspection table before and after a meal, and a camera positioned above the table photographs the food served on or remaining on the table. The image data of the food photographed is then transferred from the camera via a signal cable or the like to an eating situation estimation device, which then estimates the eating situation of the meal based on the pre-meal and post-meal food image data.

[0024] (2) Eating status estimation device The eating status estimation device is configured, for example, by a personal computer used by staff such as a medical professional or a nutritional manager. The eating status estimation device CS may also be configured, for example, by a server computer system installed on the cloud or the web.

[0025] 2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the eating status estimation device CS, respectively.

[0026] The eating condition estimation device CS includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, a sensor interface (hereinafter, interface will be abbreviated as I / F) unit 4, and an input / output I / F unit 5 are connected to the control unit 1 via a bus 6.

[0027] A camera CM is connected to the sensor I / F unit 4 via a signal cable, etc. The sensor I / F unit 4 receives image data of food output from the camera CM and converts it into an image format that the control unit 1 can process.

[0028] An input device IN using a keyboard, mouse, etc., and a display device DP using a display, etc., are connected to the input / output I / F unit 5. The input / output I / F unit 5 receives instructions and commands entered by the staff member at the input device IN, and outputs information generated by the control unit 1 that represents the estimation result of the meal intake status to the display device DP.

[0029] Note that, as a means of connection between the sensor I / F unit 4 and the camera CM, and between the input / output I / F unit 5 and the input device IN and the display device DP, instead of using signal cables, a low-power wireless interface such as Bluetooth (registered trademark) may be used. Using a wireless interface makes it possible to simplify the system configuration.

[0030] The program storage unit 2 is, for example, a combination of a non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores application programs necessary to execute various processes related to one embodiment of the present invention, in addition to middleware such as an OS (Operating System).

[0031] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, and a volatile memory such as RAM (Random Access Memory), and its storage area includes a food image storage unit 31, an edible part image storage unit 32, an area estimation model storage unit 33, a density estimation model storage unit 34, a remaining amount classification model storage unit 35, an eating status correction model storage unit 36, and an estimated information storage unit 37.

[0032] The food image storage unit 31 is used to store each of the pre-meal and post-meal food image data output from the camera CM.

[0033] The edible portion image storage unit 32 is used to store images of edible portions extracted from the pre-meal and post-meal food image data.

[0034] Model data of the area estimation model is stored in area estimation model storage unit 33. The area estimation model is composed of a machine learning model such as a neural network, and is trained in advance so that when the images of each edible part before and after eating are input as explanatory variables, the model outputs an estimated value of the amount of food eaten or the amount remaining after eating according to the area difference or area ratio of each edible part, that is, an estimated value of the eating situation, as a dependent variable.

[0035] Model data of the density estimation model is stored in the density estimation model storage unit 34. The density estimation model is also created using a machine learning model such as a neural network, and is trained in advance so that when an image of an edible part is input as an explanatory variable, a feature quantity representing the density of food in the edible part is output as a response variable.

[0036] The remaining amount classification model storage unit 35 stores model data for the remaining food amount classification model. The remaining amount classification model is created using, for example, a machine learning model, and when an image of an edible portion after eating is input, it clusters the remaining food amount status. It then outputs a classification result indicating either "completely eaten," "not eaten," or "partially eaten."

[0037] The eating condition correction model storage unit 36 ​​stores model data for the eating condition correction model. The eating condition correction model is configured, for example, by a regression model using machine learning. The eating condition correction model is trained in advance so that when an estimated value of the eating condition of a meal output from the edible portion area estimation model, a feature representing the density of food in the edible portion output from the density estimation model, and a classification result of remaining food amount output from the remaining amount classification model are input, the eating condition correction model outputs an estimated value of the eating condition of a meal corrected based on the feature representing density and the classification result of remaining food amount.

[0038] The estimated information storage unit 37 stores the estimated value of the eating state of the meal after correction estimated by the eating state correction model in association with the eater ID.

[0039] The control unit 1 has the following processing functions necessary to implement one embodiment of the present invention: a food image acquisition processing unit 11, an edible portion extraction processing unit 12, an edible portion area estimation processing unit 13, a food density estimation processing unit 14, a remaining amount classification processing unit 15, an eating status correction processing unit 16, and an eating status output processing unit 17.

[0040] The food image acquisition processing unit 11 acquires image data of food taken by the camera CM before and after eating for each patient or care recipient via the sensor I / F unit 4. Then, the food image acquisition processing unit 11 stores the acquired pre-meal and post-meal food image data as a pair in the food image storage unit 31, for example, in association with the identification information of the patient or care recipient (hereinafter referred to as the eater ID).

[0041] The edible part extraction processing unit 12 extracts images of edible parts from each of the pre-meal and post-meal food image data stored in the food image storage unit 31, and stores each of the extracted edible part images as a pair in the edible part image storage unit 32 in association with the eater's ID.

[0042] The edible portion area estimation processing unit 13 obtains an estimated value of the eating state based on the area difference or area ratio between the images of the edible parts before and after eating, using the area estimation model stored in the area estimation model storage unit 33. The edible portion area estimation processing unit 13 then passes the estimated value of the eating state to the eating state correction processing unit 16.

[0043] Food density estimation processor 14 obtains food density features represented by the pre-meal and post-meal images of each edible part using the density estimation model stored in density estimation model storage unit 34. Food density estimation processor 14 then passes the food density features to eating condition correction processor 16.

[0044] The remaining amount classification processing unit 15 obtains a classification result indicating whether the remaining amount state of the food represented by the image of the edible portion after eating is "completely eaten," "not eaten," or "partially eaten," using the remaining amount classification model stored in the remaining amount classification model storage unit 35. The remaining amount classification processing unit 15 then passes the classification result of the remaining food amount state to the eating status correction processing unit 16.

[0045] The eating state correction processor 16 corrects the estimated eating state of a meal based on the area of ​​the edible part before and after eating, using the eating state correction model stored in the eating state correction model storage unit 36. Specifically, the eating state correction processor 16 corrects the estimated eating state of a meal output from the edible part area estimation model based on the feature representing the density of food in the edible part output from the density estimation model and the remaining amount classification result output from the remaining amount classification model. The corrected estimated eating state is then stored in the estimated information storage unit 37 in association with the eater's ID.

[0046] The eating condition output processing unit 17 reads out the corrected eating condition estimated value from the estimated information storage unit 37, generates display data representing the read out estimated value, and outputs the data from the input / output I / F unit 5 to the display device DP.

[0047] (Example of operation) Next, an example of the operation of the eating condition estimation device CS configured as above will be described.

[0048] FIG. 4 is a flowchart showing an example of the procedure and content of the eating status estimation process executed by the control unit 1 of the eating status estimation device CS.

[0049] (1) Acquisition of food image data and extraction of edible parts When checking the eating status of a patient or a care recipient, a medical professional or nutritionist or other staff member first places the tableware or tray containing the meal on the examination table before the meal, and in this state operates the input device IN to input an estimated instruction command.

[0050] The control unit 1 of the eating condition estimating device CS monitors the setting of a tray before a meal and the input of an estimation instruction command in step S10. When the setting of a tray before a meal and the input of an estimation instruction command are detected in this state, first in step S11, the camera CM is activated under the control of the food image acquisition processing unit 11. As a result, the camera CM captures an image of the food served on the tray.

[0051] The food image acquisition processing unit 11 may detect that the pre-meal tray has been set on the inspection table and activate the camera CM. In this case, there is no need to input an estimation instruction command.

[0052] The food image acquisition processing unit 11 captures the image data of the food captured by the camera CM via the sensor I / F unit 4, and stores the captured image data of the food in the food image storage unit 31 in association with the eater ID. The eater ID may be manually input by a staff member via the input device IN, or a two-dimensional code representing the eater ID displayed on the tray may be recognized from the image captured by the camera CM.

[0053] Next, in step S12, the control unit 1 of the eating state estimation device CS, under the control of the edible part extraction processor 12, reads the pre-eating food image data from the food image storage unit 31 and extracts images of edible parts from the read food image data. The edible parts can be extracted, for example, based on the color or shape of the food. For example, the edible part extraction processor 12 colors only the extracted images of the edible parts from the food image data with a predetermined color that is different from the color of the tableware, and stores this colored food image data in the edible part image storage unit 32 in association with the eater's ID.

[0054] After finishing the meal, the staff member places the tableware or tray on the inspection table, and in this state operates the input device IN to input an inferential instruction command.

[0055] When the control unit 1 of the eating condition estimation device CS detects in step S13 that a post-meal tray has been set on the inspection table and that an estimation instruction command has been input, it acquires post-meal food image data captured by the camera CM in step S14 under the control of the food image acquisition processing unit 11, and then extracts edible parts from the post-meal food image data in step S15 under the control of the edible part extraction processing unit 12. The process of acquiring post-meal food image data and the process of extracting edible parts are performed in the same way as in the case of pre-meal food described above.

[0056] (2) Estimation of contact state based on edible area Once the process of acquiring pre-meal and post-meal food image data and the process of extracting edible parts are completed, the control unit 1 of the eating status estimation device CS first executes a process of estimating the eating status of a meal based on the edible part area under the control of the edible part area estimation processing unit 13 in step S16 as follows.

[0057] That is, edible portion area estimation processing unit 13 reads images of each edible portion before and after eating from edible portion image storage unit 32 and inputs the read images of each edible portion into an area estimation model stored in area estimation model storage unit 33. The area estimation model calculates the area of ​​the region surrounding each edible portion for each of the input pre- and post-meal edible portion images and calculates the difference or area ratio between the calculated areas. Then, based on the calculated area difference or area ratio, it calculates the amount of food intake or the amount remaining after eating and outputs the calculated amount of food intake or the amount remaining after eating as an estimate of the eating situation.

[0058] For example, suppose that food images before and after eating are obtained as shown in Figures 5(a) and (b), where FD1 and FD2 represent the edible parts before and after eating, respectively, and DS represents the edge of the dish.

[0059] In this case, the area estimation model sets rectangular areas E1 and E2 surrounding the edible portion, as shown in Figures 5(a)' and 5(b)'. The area of ​​the set rectangular areas E1 and E2 is then calculated, for example by counting the number of pixels, and the difference (E1-E2) or area ratio (E2 / E1) between the calculated rectangular areas E1 and E2 is calculated. The area estimation model calculates the amount of food eaten or the amount remaining after eating based on the calculated area difference or area ratio, and outputs the calculated amount of food eaten or the amount remaining after eating as an estimate of the eating situation.

[0060] The edible portion area estimation processing unit 13 obtains an estimated value of the eating state from the area estimation model, and passes the obtained estimated value of the eating state to the eating state correction processing unit 16.

[0061] (3) Estimation of food density Next, in step S17, the control unit 1 of the eating condition estimating device CS executes the food density estimation process as follows under the control of the food density estimation processing unit .

[0062] That is, food density estimation processing unit 14 reads out images of each edible part before and after eating from edible part image storage unit 32, and inputs the read-out images of each edible part into a density estimation model stored in density estimation model storage unit 34. In response to this, the density estimation model calculates feature quantities representing food density (degree of dispersion) from the input images of each edible part before and after eating, and passes the calculated density feature quantities to eating state correction processing unit 16.

[0063] For example, suppose we have obtained food images of before and after eating, as shown in Figures 6(a) and 6(b). In this case, the density estimation model calculates the density of foods FD3 and FD4 in the edible area, i.e., the feature that represents the degree of dispersion. In this example, it is detected that the density of food FD4 after eating is lower than that of food FD3 before eating.

[0064] (4) Classification of food remaining amount Next, in step S18, the control unit 1 of the eating condition estimating device CS, under the control of the remaining amount classification processing unit 15, classifies the state of the remaining amount of food on the tableware after the meal as follows.

[0065] That is, the remaining amount classification processing unit 15 reads out the image of the edible part after eating from the edible part image storage unit 32, and inputs the read out image of the edible part into the remaining amount classification model stored in the remaining amount classification model storage unit 35. The remaining amount classification model clusters the state of the remaining amount of food based on the input image of the edible part after eating, and outputs one of the following results: "completely eaten," "not eaten," or "partially eaten."

[0066] For example, if the input image of the edible part is an image of an uneaten part as shown in Figure 7(a), the classification result will be "uneaten" according to the probability of the amount of food eaten as shown in (a)'. If the input image of the edible part is an image of a fully eaten part as shown in Figure 7(b), the classification result will be "fully eaten" according to the probability of the amount of food eaten as shown in (b').

[0067] The remaining amount classification processing unit 15 passes the classification result of the remaining amount state output from the remaining amount classification model to the eating state correction processing unit 16.

[0068] (5) Correction of estimated feeding status Next, in step S19, the control unit 1 of the eating status estimation device CS, under the control of the eating status correction processing unit 16, corrects the estimated value of the eating status of the meal based on the area of ​​the edible part obtained by the edible part area estimation processing unit 13 using the eating status correction model as follows.

[0069] That is, eating state correction processing unit 16 inputs the eating state estimate value of the meal output from the edible portion area estimation model, together with the feature amount representing the density of food in the edible portion output from the density estimation model and the classification result of the remaining amount output from the remaining amount classification model, into the eating state correction model. The eating state correction model corrects the input eating state estimate value of the meal based on the feature amount representing the density of food in the edible portion and the classification result of the remaining food amount after eating.

[0070] For example, as shown in Figures 6(a)' and 6(b)', even if the area estimation model detects almost no difference between the areas E3 and E4 of the edible parts before and after eating, the density estimation model estimates that the feature value representing the food density (degree of dispersion) of the edible part FD4 after eating is lower than that of the edible part FD3 before eating. Therefore, the edible part area estimation model corrects the estimated value of the eating situation obtained based on the edible part areas E3 and E4 to increase the amount of food intake (reduce the amount remaining after eating) based on the feature value representing the food density (degree of dispersion) of the edible part FD4 obtained by the density estimation model.

[0071] As a result, even if the area of ​​the edible part does not change significantly between before and after eating, but the height of the food does, the estimated eating status can be corrected based on the characteristics of the food density (degree of dispersion).

[0072] Furthermore, for example, even if there is no food left on the dish after a meal but sauce remains, and this sauce is detected as the area of ​​the edible portion, the remaining food amount classification model will classify the remaining food amount as "completely eaten." Therefore, even if the estimated eating status based on area indicates that the amount of food eaten is small (a large amount of food remaining), the estimated eating status is corrected to indicate that the amount of food eaten is large (a small amount of food remaining) based on the classification result of the remaining food amount. This makes it possible to estimate eating status with higher accuracy than when estimating based on area alone.

[0073] The eating state correction processing unit 16 stores the corrected eating state estimated value output from the eating state correction model in the estimated information storage unit 37 in association with the eater ID.

[0074] (6) Output of feeding status estimation results Finally, in step S20, the control unit 1 of the eating condition estimation device CS reads out the estimated value of the eating condition of the eater together with the eater ID from the estimated information storage unit 37 under the control of the eating condition output processing unit 17, and generates display data based on the read information. The eating condition output processing unit 17 then outputs the generated display data from the input / output I / F unit 5 to the display device DP. Thus, the estimated value of the eating condition of the eater is displayed on the display device DP.

[0075] The eating status output processing unit 17 may output the estimated value of the eating status of the eater to a printer, for example, to print it out, or may output it to an external storage medium such as a USB memory, for example, to store it.

[0076] (effect) As described above, in one embodiment, images of edible parts are extracted from the pre-meal and post-meal food image data acquired from the camera CM, and an estimate of the eating status is obtained based on the difference in area or area ratio of the extracted images of the edible parts. At the same time, features representing food density are calculated based on the pre-meal and post-meal images of the edible parts, and the remaining food amount is classified based on the post-meal image of the edible parts. The estimated eating status is then corrected based on the classification results of the features representing food density and the remaining food amount, and the corrected estimated eating status is output.

[0077] Therefore, according to one embodiment, for example, in cases where the area of ​​the edible part does not change significantly between before and after eating, but the height of the food does change, the estimated result of the eating status is corrected based on the density of the food, making it possible to estimate the eating status with higher accuracy than when estimating based on area alone.

[0078] Furthermore, even if after a meal there is no food left on the dish but sauce remains and this sauce is detected as the area of ​​the edible portion, the amount of food remaining is classified as "completely eaten," and the eating status estimation result based on area is corrected based on this classification result. This makes it possible to estimate eating status with higher accuracy than when estimating based on area alone.

[0079] [Other embodiments] (1) In one embodiment, the process of estimating the level of eating based on the area of ​​edible parts, the process of estimating features representing food density, the process of classifying remaining portions, and the process of correcting the estimated value of the eating status of a meal are all performed using a machine learning model. However, it is not necessary to use a machine learning model.

[0080] (2) In one embodiment, the case where one type of food is served on a dish has been described as an example, but this invention can also be applied to cases where multiple types of food are served on a dish. In this case, the multiple types of food served on a single dish are recognized and separated based on the image data, and the eating status of each separated food is estimated. In this way, it is possible to estimate the eating status of each food.

[0081] (3) In one embodiment, an example was given of estimating the eating status based on the area of ​​the edible part. However, it is also possible to obtain three-dimensional information of food before and after eating based on multiple image data captured using a stereo camera or image data including depth information captured using a depth camera, and estimate the eating status based on each of the obtained three-dimensional data.

[0082] (4) The functional configuration, processing procedures, and processing contents of the eating status estimation device can be modified in various ways without departing from the spirit of the present invention.

[0083] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0084] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0085] CS: Eating status estimation device CM...camera IN...input device DP: Display device 1...Control unit 2...Program memory section 3...Data storage unit 4...Sensor I / F section 5...Input / output interface 6...Bus 11...Food image acquisition processing unit 12...Edible part extraction processing section 13...Edible area estimation processing unit 14...Food density estimation processing unit 15...Remaining amount classification processing unit 16...Eating status correction processing unit 17...Eating status output processing unit 31...Food image memory unit 32...Edible part image storage section 33...Area estimation model memory unit 34...Density estimation model memory section 35...Remaining amount classification model memory unit 36...Eating status correction model memory section 37… Estimated information storage unit

Claims

1. a first processing unit that acquires a first food image captured before eating and a second food image captured after eating the food; a second processing unit that extracts a first edible portion before eating from the first food image and a second edible portion after eating from the second food image; A third processing unit that estimates the eating status of the food based on a change in area between the first edible portion and the second edible portion; a fourth processing unit that estimates the density of the food in the first edible portion and the second edible portion; a fifth processing unit that corrects the estimation result of the food intake state based on the density of the food; An eating status estimation device comprising:

2. Further provided is a sixth processing unit that classifies the remaining amount of food from the second edible portion as being completely eaten, not eaten, or partially eaten, The fifth processing unit corrects the estimation result of the food intake state based on the classification results of the density and the remaining amount state of the food. The eating status estimation device according to claim 1 .

3. 2. The eating status estimation device of claim 1, wherein the third processing unit obtains an estimation result of the food eating status using a machine learning model that receives an image representing the first edible part and an image representing the second edible part as input and outputs an estimation result of the food eating status based on an amount of change in area between the first edible part and the second edible part.

4. 2. The eating status estimation device of claim 1, wherein the fourth processing unit obtains the feature values ​​representing the density of the food using a machine learning model that receives an image representing the first edible part and an image representing the second edible part as input and outputs feature values ​​representing the density of the food in the first edible part and the second edible part.

5. The eating status estimation device of claim 2, wherein the sixth processing unit obtains a classification result of the remaining food amount state using a machine learning model that takes an image of the second edible portion as input, classifies the remaining food amount state into either completely eaten, not eaten, or partially eaten, and outputs the classification result.

6. An eating status estimation method executed by an information processing device, comprising: acquiring a first food image taken before eating food and a second food image taken after eating food; extracting a first edible part before eating from the first food image and a second edible part after eating from the second food image; A step of estimating the eating status of the food based on a change in area between the first edible portion and the second edible portion; estimating the density of the food in the first edible portion and the second edible portion; correcting the estimated result of the food intake state based on the density; The feeding status estimation method includes the steps of:

7. 3. A program causing a processor included in the eating status estimation device to execute at least one of the processes executed by each processing unit included in the eating status estimation device according to claim 1 or 2.

Images