Food behavior sensing device

Abstract

To provide an eating behavior sensing system that can grasp the order in which foods are ingested.SOLUTION: An eating behavior sensing system of the present invention comprises: contact detection means for detecting when foods have come into contact with eating utensils; an imaging unit for imaging the foods arranged in front of an eater; and food identification means for identifying a type of food ingested for each bite based on a first image captured by the imaging unit when contact with the food is detected by the contact detection means and pre-stored arrangement information of the foods.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an eating behavior sensing system. [Background technology]

[0002] Various sensing systems have been proposed to grasp behavior during meals, particularly what foods and how much have been ingested. For example, claim 1 of Patent Document 1 describes how "images of one person's pre-meal portion with food on each dish on a tray are taken in both the pre-meal and post-meal states" and how "only the food portion on each dish image is extracted, the amount of change in the food portion before and after the meal is calculated in terms of area, and the calorie intake and various nutrients such as vitamins are calculated in reference to a food database created in advance." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-70908 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technique described in Patent Document 1, it is difficult to grasp the order in which foods are ingested.

[0005] An object of the present invention is to provide an eating behavior sensing system capable of grasping the order in which foods are ingested. [Means for solving the problem]

[0006] The present invention has been made in consideration of the above-mentioned problems, and the eating behavior sensing system of the present invention comprises a contact detection means for detecting when food has come into contact with eating utensils, an imaging unit for imaging the food arranged in front of the eater, and a food identification means for identifying the type of food eaten for each bite based on a first image acquired by the imaging unit when contact with the food is detected by the contact detection means and pre-stored information about the food arrangement. Effect of the Invention

[0007] According to the present invention, it is possible to provide an eating behavior sensing system capable of grasping the order in which foods are ingested. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an overview of an eating behavior sensing system. [Diagram 2] FIG. 2 is a diagram showing the overall configuration of eating utensils. [Diagram 3] FIG. 1 is a functional block diagram showing the configuration of an eating behavior sensing system. [Figure 4] FIG. 13 is a diagram showing an overview of an eating behavior sensing system according to a modified example. [Diagram 5] 11A to 11C are diagrams showing changes in images captured during a taking action of eating one bite. [Figure 6] 1 is a flowchart of an eating behavior sensing system. [Figure 7] FIG. 7 is a diagram for explaining a modification of step S3 in FIG. 6. [Figure 8] 13A to 13C are diagrams showing changes in images captured when a person eats while holding the container up. [Figure 9] FIG. 11 is a flowchart showing a process for determining whether or not a reference image can be updated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] FIG. 1 is a diagram showing an overview of an eating behavior sensing system. As shown in FIG. 1, the eating behavior sensing system is composed of an imaging and processing device 100, eating utensils 101, and a terminal device 102 (user interface). A tray 50 on a table has a plurality of containers 60, such as plates and bowls, for holding food (dishes). The imaging and processing device 100 has an imaging unit such as a camera, and is installed on the ceiling together with a lighting device, for example, so that all the food on the tray 50 fits within the angle of view. The eating utensils 101 include cutlery such as a spoon and a fork as well as chopsticks, but the following description will be given taking a spoon as an example.

[0011] Fig. 2 is a diagram showing the overall configuration of the eating utensil. As shown in Fig. 2, the eating utensil 101 (spoon) is composed of a head part 40 (cup) that supports food and a handle part 41 (grip) that is held by the fingers.

[0012] The head 40 is connected to a load cell built into the handle 41, and the force applied to the head 40 can be measured by the load cell. The head 40 is detachable from the load cell in the handle 41, and the head 40 can also be removed for cleaning. When a capacitance sensor is used as the contact detection means as described below, the head 40 is preferably made of a conductive material (e.g., metal). Meanwhile, the handle 41 has built-in posture detection means, a calculation unit, a communication unit, and a battery, which will be described later.

[0013] 3 is a functional block diagram showing the configuration of the eating behavior sensing system. As described above, the eating behavior sensing system includes the eating utensils 101, the image capturing and processing device 100, and the terminal device .

[0014] As shown in FIG. 3, the eating utensil 101 has a sensor unit 10, a calculation unit 20, a sensor information storage unit 21, a communication unit 22, and a battery 23.

[0015] The sensor unit 10 includes a contact detection means 11, a posture detection means 12, and a weight measurement means 13. The contact detection means 11 detects when food or a human body (mainly the mouth) comes into contact with the head unit, and is, for example, a capacitance sensor. The posture detection means 12 detects the posture (tilt) of the eating utensil 101, and is, for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, or the like. The weight measurement means 13 measures the weight of the food supported by the head unit, and in this embodiment, a load cell is used, but something other than a load cell may be used.

[0016] The calculation unit 20 performs a predetermined calculation based on the information input from each sensor of the sensor unit 10, and is preferably a controller with relatively low power consumption, such as a microcomputer. For example, the calculation unit 20 monitors the measured value of the contact detection means 11, and when a predetermined change occurs, it can detect that the head unit has come into contact with food, that the head unit has come into contact with the mouth of the eater (user), and the like. In addition, when the head unit comes into contact with the mouth, the change in capacitance is larger than when the head unit comes into contact with food, so that the contact object can be distinguished by providing a different threshold value. In addition, the calculation unit 20 also acquires the weight of the food for each bite (one time of eating), triggered by the detection information of the contact with food. At this time, the calculation unit 20 accurately calculates the weight of the food by, for example, correcting the vertical force among the forces measured by the load cell using the information on the posture detected by the posture detection means 12. In addition, the calculation unit 20 is equipped with a time measurement means (not shown) and can calculate the time from the start of the meal, as well as the start time and end time of the meal.

[0017] The sensor information storage unit 21 records the detection information by the contact detection means 11, and information calculated by the calculation unit 20, such as the weight of each bite of food and the start time of eating. The communication unit 22 transmits information, such as the timing of contact with the food and the timing of contact with the mouth, and the weight of the food, to the imaging processing device 100 for each bite. The communication unit 22 is capable of wireless communication connection such as Bluetooth (registered trademark) and wireless LAN, and transmits and receives information at regular time intervals. The battery 23 is built into the handle and supplies power to drive the sensors of the sensor unit 10, the calculation unit 20, and the communication unit 22. It is preferable that the battery be a rechargeable battery since repeated use is expected, but a dry cell may also be used.

[0018] The imaging processing device 100 is a device that has both the function of imaging food and the function of performing predetermined calculations including image processing, and specifically, as shown in Figure 3, has a communication unit 32, an imaging unit 35, an image storage unit 37, a food information database 34, a food identification means 30, and a meal information storage unit 31.

[0019] The communication unit 32 receives information from the communication unit 22 of the eating utensils 101 and transmits information processed by the image capturing and processing device 100 to the terminal device 102. The image capturing unit 35 captures images of the food items arranged on a table or the like in front of the eater, as well as the eater's fingers and eating utensils 101. From the viewpoint of image processing, it is preferable that the eater's face, etc., does not enter the camera's field of view, but images may be captured to leave a record of the eater. The image storage unit 37 stores images including food captured by the imaging unit 35.

[0020] Food information database 34 stores food arrangement information related to the position (center coordinates) of each food in front of the eater. Here, food arrangement information can also be acquired by analyzing an image captured by imaging unit 35 and stored in image storage unit 37 before the start of a meal. In addition, when meals are provided at a facility, the position of each food may be determined for each menu. In that case, the position of each food may be acquired automatically from the menu without using imaging unit 35.

[0021] The food identification means 30 identifies the type of food eaten for each bite based on the image captured by the imaging unit 35, the information received from the eating utensils 101 via the communication unit 32, and the food arrangement information stored in the food information database 34. Specifically, the food identification means 30 first calculates the eating target position by comparing the first image captured by the imaging unit 35 when contact with the food is detected by the contact detection means 11 with a reference image captured by the imaging unit 35 before the contact and stored in the image storage unit 37. Next, the food identification means 30 identifies the type of food corresponding to the eating target position by collating the eating target position with the food arrangement information stored in the food information database 34.

[0022] The meal information storage unit 31 records information such as the type of food identified by the food identification means 30, and the weight of the food received from the eating utensils 101 via the communication unit 32, for each bite.

[0023] The terminal device 102 is, for example, a tablet terminal including a smartphone in a home, and is, for example, a PC in a facility including a restaurant or a hospital. As shown in FIG. 3, the terminal device 102 has a communication unit 36 ​​and a display unit 33. The communication unit 32 receives information from the communication unit 32 of the imaging processing device 100. The display unit 33 displays the progress of the ingested food on a dashboard for each bite based on the information received via the communication unit 32. The display unit 33 may also display the pace of eating using meal time information calculated by the eating utensil 101. In addition, if the pace of eating is faster than a certain condition, the display unit 33 may display a message encouraging the user to eat slowly.

[0024] In the example described above using Fig. 1 and Fig. 3, the image capturing and processing device 100 and the terminal device 102 are separate, but they may be integrated. Fig. 4 is a diagram showing an outline of an eating behavior sensing system according to a modified example. In this modified example, the processing device 104 has both the functions of the image capturing and processing device 100 and the terminal device 102, and as shown in Fig. 4, an image capturing unit 35 and a display unit 33 are installed in one housing constituting the processing device 104. The processing device 104 is disposed so as to face the eater with food between them, for example, and the image capturing unit 35 is positioned above the container 60 that contains the food, so that it is possible to capture an image of the food and eating utensils 101 arranged in front of the eater.

[0025] The display unit 33 can also inform the eater of data such as weight and body fat obtained from devices such as a weight scale and a health meter, as well as the average eating pace and average meal start time, before and after a meal, and can also alert the eater to the way he or she eats. Such notifications allow the eater to recognize the correlation between the way he or she eats and the biomarkers, and it is expected that the eater will continue to use the system for a long time. In addition, if the eater is to measure his or her blood glucose level a certain time after finishing a meal, the display unit 33 can prevent the eater from forgetting to measure by prompting the eater to measure after the certain time has passed. Furthermore, the display unit 33 may inform the eater of the order of eating, eating pace, meal time, etc., suitable for suppressing a sudden rise in blood glucose level. These notifications may be sent not only to the display unit 33 but also to a smartphone, a smartwatch, etc. via the communication unit.

[0026] 5 is a diagram showing changes in images captured during the action of taking a bite of food. From the start to the end of the meal, the changes in images shown in FIG. 5 are repeated for each bite.

[0027] (1) is an image taken before or at the start of a meal. Food A 52 and food B 53 placed on a tray 50 are visible within the camera's field of view 51, but the eating utensils 101 and the eater's hands are not. This image is stored in the image storage unit 37 as a reference image to be used in subsequent processing. Furthermore, the type of each food and its center coordinates are obtained by analysis using this image, and are stored in the food information database 34 as food arrangement information.

[0028] (2) is an image taken at the moment when the meal begins and the eating utensils 101 come into contact with food A52. In addition to food A52 and food B53, the eating utensils 101 and the eater's hand 55 are also captured within the camera's field of view 51. The image taken at this time is compared with the reference image by the food identification means 30 as the first image.

[0029] (3) is a result of comparing the reference image with the first image, and filling in a difference region 56 having a difference equal to or greater than a first threshold value. The difference region 56 corresponds to the eating utensil 101 and the hand 55, and the point among them that is the furthest from the eater can be regarded as the eating target position 57.

[0030] (4) is an image of the moment when the eating utensil 101 is placed in the eater's mouth 58. From the image (4) (second image), it can be seen that there is a little less food A52 compared to the image (1) (reference image). The food identification means 30 updates the reference image stored in the image storage unit 37 with the second image at this time.

[0031] Here, the amount of food eaten is determined for each bite by the food identification means 30 based on the weight information measured by the weight measurement means 13. The amount of food may also be determined by the food identification means 30 based on the difference (area reduction) between the reference image (1) and the second image (4) and food density information stored in advance in the food information database 34 or the like. The latter case is less accurate than the former case, but has the advantage that the amount of food can be determined even when the eating utensil 101 is not equipped with the weight measurement means 13.

[0032] Furthermore, the type of food eaten is identified by the food identification means 30 by comparing the eating target position 57 with the food arrangement information stored in the food information database 34. Specifically, the food identification means 30 calculates the distance between the eating target position 57 and the central coordinates of each food included in the food arrangement information, and estimates that the food with the closest distance is the food to be eaten.

[0033] FIG. 6 is a flowchart of the eating behavior sensing system.

[0034] Before or at the start of a meal, the imaging unit 35 captures an image of the food items arranged in front of the eater, and the image is stored in the image storage unit 37 as a reference image (step S1).

[0035] Next, the food identification means 30 judges whether there is detection information indicating that the eating utensil 101 has come into contact with food (step S2). If there is no detection information of food contact, step S2 is repeated until there is detection information. As described above, the detection information of food contact is transmitted from the contact detection means 11 of the eating utensil 101 to the imaging processing device 100 via the communication unit 22.

[0036] In step S2, if there is detection information of food contact, the food identification means 30 calculates the difference between the first image (current image) captured by the imaging unit 35 at that time and the reference image captured in step S1, and determines the point farthest from the eater within the difference area equal to or greater than the first threshold as the eating target position (step S3). Note that if the eating utensil 101 and the food repeatedly come into contact with and separate from each other multiple times before the eater brings the food to his / her mouth, multiple farthest points may be calculated, and in that case, for example, the average of the multiple points is determined as the eating target position.

[0037] Next, the food identification means 30 judges whether there is detection information indicating that the eating utensils 101 have come into contact with the human body (mouth) of the eater (step S4). If there is no detection information of human body contact, step S4 is repeated until there is detection information. Note that, like the detection information of food contact, the detection information of human body contact is also transmitted from the contact detection means 11 of the eating utensils 101 to the imaging processing device 100 via the communication unit 22.

[0038] In step S4, if there is human body contact detection information, the food identification means 30 calculates the distance between the eating target position and the center coordinates of each food included in the food arrangement information stored in the food information database 34 (step S5). After that, the food identification means 30 estimates the type of food that is closest in distance as the type of food that has been eaten (step S6). In addition, the food identification means 30 updates the reference image captured in step S1 using the second image captured by the imaging unit 35 when there is human body contact detection information (step S7).

[0039] Next, the food identification means 30 again judges whether there is any detection information indicating that the eating utensil 101 has come into contact with food (step S8). If there is no detection information of food contact even after a predetermined time has passed, the meal is deemed to have ended. On the other hand, if there is detection information of food contact in step S8, the meal is deemed to be continuing, and the process returns to step S3, and the subsequent processes are repeated.

[0040] By the above-mentioned process, the type of food ingested can be identified for each bite and displayed in real time on the display unit 33. In addition, since the type of food ingested is stored in the meal information storage unit 31 for each bite, it is also possible to grasp the order in which the food is ingested. Furthermore, since the type of food is identified by a method in which the difference from a reference image is calculated and compared with food arrangement information, triggered by detection information of contact between the eating utensils 101 and the food or human body, the computation load is smaller than that of image processing using a neural network. Although omitted in the flowchart of FIG. 6, the weight of the food ingested is also identified for each bite, so the pace of eating can also be grasped.

[0041] Further, the eating utensil 101 may be provided with a temperature sensor, an odor sensor, a salt sensor, a pulse wave sensor, etc., as a sensor mounted thereon. For example, by providing a temperature sensor such as a radiation thermometer capable of non-contact measurement in the head, it is possible to determine the presence or absence of food by temperature, and to measure the temperature of the food and the body temperature of the eater. In addition, a smell sensor that detects the smell of food may be provided in the handle, and the smell data may also be recorded in the meal information storage unit 31 of the image capture processing device via the communication unit, which may be useful for reproducing the dish. For example, the components of spices and seasonings may be acquired as smell data to estimate the taste of the dish, and the components of putrid smells may be acquired as smell data to determine whether the dish is edible. Furthermore, by providing a salt sensor that measures the salt content of food in the handle, it is possible to use it as a guide for seasoning and to prevent excessive salt intake. In addition, a pulse wave sensor that measures the pulse wave of the human body may be provided in the handle, and changes in the pulse wave of the eater while eating may be output to the display unit 33. For example, if food containing a lot of carbohydrates is ingested on an empty stomach, causing a sudden rise in blood sugar level and abnormalities in the pulse wave, an alarm may be output to the display unit 33 urging the user to eat other food.

[0042] Fig. 7 is a diagram for explaining a modification of step S3 in Fig. 6. (1) is the first image (current image) at the moment when the eating utensil 101 comes into contact with the food, and (2) is the result of comparing the reference image with the first image, in which the difference region 56 having a difference equal to or greater than the first threshold value is filled in.

[0043] In the case of step S3 in Fig. 6, the point in the differential region 56 that is farthest from the eater is determined as the eating target position, so position 57A in Fig. 7(2), i.e., the position of the eater's fist, may be determined as the eating target position. However, the actual eating target position is position 57B of the tip of the eating utensil 101, which leads to a decrease in the accuracy of identifying the food.

[0044] Therefore, in this modification, the side with which the contactor holds the eating utensil 101, i.e., the dominant hand of the eater (right-handed in the example of FIG. 7), can be input in advance by an input unit (not shown) of the terminal device 102. Then, based on the input information on the dominant hand, the food identification means 30 determines the point furthest from the shoulder on the dominant hand side (position 57B in the example of FIG. 7) as the eating target position. This improves the accuracy of identifying the food. The position of the eater's shoulder may be determined in advance from the relative positions of the table and chair, or may be determined by skeletal recognition using an image captured by the imaging unit 35.

[0045] FIG. 8 is a diagram showing changes in an image captured when a person eats while holding a container upright.

[0046] (1) is an image taken before or at the start of a meal. Food A 52 and food B 53 placed on a tray 50 are visible within the camera's field of view 51, but the eating utensils 101 and the eater's hands are not. This image is stored in the image storage unit 37 as a reference image to be used in subsequent processing. Furthermore, the type of each food and its center coordinates are obtained by analysis using this image, and are stored in the food information database 34 as food arrangement information.

[0047] (2) is an image taken at the moment when the eating utensil 101 comes into contact with food B53 outside the camera's field of view 51 after the eater has started to lift the container 60 containing food B53. Food A52 is visible within the camera's field of view 51, but food B53 is not. The image taken at this time is compared with the reference image by the food identification means 30 as the first image.

[0048] (3) is a result of comparing the reference image with the first image, and filling in a difference region 56 having a difference equal to or greater than a first threshold value. The difference region 56 corresponds to a container 60, and the point in the difference region that is the furthest from the eater is regarded as an eating target position 57.

[0049] (4) is an image taken at the moment when the eating utensil 101 is placed in the eater's mouth 58. If the reference image were updated with the image taken at this time (second image), the food B53 would not be reflected within the field of view 51 of the reference image. Therefore, even if an image taken at the moment when the eating utensil 101 comes into contact with the food B53 again for the next bite is compared with this reference image, no appropriate difference is obtained, making it difficult to calculate the eating object position.

[0050] Therefore, it is desirable to provide a process for determining whether or not the reference image can be updated in step S7 in Fig. 6. Fig. 9 is a diagram showing a flow for determining whether or not the reference image can be updated.

[0051] First, the food identification means 30 acquires a second image captured by the imaging unit 35 when there is human contact detection information as a candidate for a reference image (step S7A). Next, the food identification means 30 determines whether the matching rate between the second image and the reference image at that time is equal to or higher than a second threshold (step S7B). If the matching rate is equal to or higher than the second threshold, the food identification means 30 determines that the container has not been lifted and updates the reference image using the second image (step S7C).

[0052] On the other hand, if the matching rate is less than the second threshold in step S7B, the food identification means 30 determines that the container is being lifted, and does not update the reference image (step S7D).

[0053] In this way, by providing a process for determining whether or not to update the reference image, it is possible to accurately identify the type of food eaten, even when, for example, Japanese food is being eaten, in which the container is frequently moved.

[0054] A weight sensor may be provided on the underside of the tray 50, or the tray 50 may be placed on the weight sensor, and the container 60 for containing food may be placed on the tray 50. In this case, a weight measuring means may be used to measure the weight of the tray 50 using the contact of the eating utensil 101 with the mouth of the eater (user) as a trigger (starting point), detect a change in weight, and record the intake amount for each bite. In this case, even if the container 60 for containing food contains a liquid such as miso soup, the intake amount can be recorded with high accuracy.

[0055] The present invention is not limited to the above-mentioned embodiment, and various modifications are possible. For example, in the above-mentioned embodiment, the trigger for calculating the eating target position, i.e., the trigger for comparing the captured image (first image) with the reference image, is the timing when contact with the food is detected, but instead, it may be the timing when separation from the food is detected. In addition, if a display (e.g., a mark) or a shape (e.g., a partition or a groove) that specifies the placement location of the container 60 is provided in advance on the tray 50, the position of the food does not change significantly with respect to the food placement information, and the accuracy of identifying the food is further improved. [Explanation of symbols]

[0056] 10...sensor unit, 11...contact detection means, 12...posture detection means, 13...weight measurement means, 20...calculation unit, 21...sensor information storage unit, 22...communication unit, 23...battery, 30...food identification means, 31...meal information storage unit, 32...communication unit, 33...display unit, 34...food information database, 35...imaging unit, 36...communication unit, 37...image storage unit, 40...head unit, 41...handle unit, 50...tray, 51...field of view, 52...food A, 53...food B, 55...hand, 56...difference area, 57...eating target position, 58...mouth, 60...container, 100...imaging processing device, 101...eating utensils, 102...terminal device, 104...processing device

Claims

1. A contact detection means for detecting when food comes into contact with the eating utensil; An imaging unit that images the food arranged in front of a person to eat; a food identification means for identifying the type of food eaten for each bite based on a first image captured by the imaging unit when contact with the food is detected by the contact detection means and pre-stored information on the arrangement of the food; An eating behavior sensing system equipped with

2. The eating behavior sensing system according to claim 1 , The eating behavior sensing system is characterized in that the food identification means calculates the eating target position by comparing a pre-stored reference image with the first image, and identifies the type of food by matching the eating target position with the placement information.

3. The eating behavior sensing system according to claim 2, The eating behavior sensing system is characterized in that the food identification means calculates the difference between the reference image and the first image, and determines the point farthest from the eater among the difference areas equal to or greater than a first threshold value as the eating target position.

4. The eating behavior sensing system according to claim 3, The eating behavior sensing system is characterized in that the food identification means determines, as the eating target position, a point in the difference area that is farthest from the shoulder on the side where the eater holds the eating utensil.

5. The eating behavior sensing system according to claim 2, The contact detection means also detects contact of the eating utensil with the body of the eater, The eating behavior sensing system is characterized in that the food identification means updates the reference image with a second image acquired by the imaging unit when contact with the human body is detected.

6. The eating behavior sensing system according to claim 5 , The eating behavior sensing system is characterized in that when the matching rate between the second image and the reference image is less than a second threshold, the food identification means determines that the container containing the food has been lifted and does not update the reference image.

7. The eating behavior sensing system according to claim 6, An eating behavior sensing system characterized in that a mark or shape that specifies the placement location of the container is provided in advance.

8. The eating behavior sensing system according to claim 1 , The eating utensil includes the contact detection means, a weight measurement means for measuring a weight of the food supported by the eating utensil, and a communication unit for communicating with the food identification means, The eating behavior sensing system according to claim 1, wherein the food identification means identifies the amount of the food ingested for each bite based on weight information received via the communication unit.

9. The eating behavior sensing system according to claim 5 , The eating behavior sensing system is characterized in that the food identification means identifies the amount of food eaten for each bite based on the difference between the second image and the reference image and pre-stored density information of the food.

Images