Meal monitoring device, meal monitoring system, meal monitoring method, meal monitoring program, learning device and inference device

Abstract

To solve the problem that conventional calorie estimation programs sometimes perform inappropriate response because of monitoring of an object person to be monitored and health management support based on wrong dishes without correctly estimating the dishes from images.SOLUTION: According to the present disclosure, a meal monitoring device includes a meal situation estimation unit for estimating a meal situation of an object person to be monitored from bowl information including bowl images obtained by extracting a bowl area being an area of bowls detected from a dining table image of the object person to be monitored from the image and information on positions of the bowls on the dining table, and a caution needed situation determination unit for determining whether a situation of the object person to be monitored corresponds to a caution needed situation by using the meal situation and model data showing meal situations determined as normal.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a meal monitoring device, a meal monitoring system, a meal monitoring method and a meal monitoring program, a learning device, and an inference device.

Background Art

[0002] As part of the monitoring or health management of single persons or the elderly, there is a meal monitoring system for monitoring meals. One of the meal monitoring systems is a calorie estimation program for estimating the calories of the food served in a dish.

[0003] The calorie estimation program disclosed in Patent Document 1 detects the shape of the dish in which the food is served and the color of the food from the captured image of the food, and based on the detected shape of the dish and the color of the food, the food and its calories are estimated from a database in which a plurality of foods and the calories of the food are associated with the shape of the dish and the color of the food.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the calorie estimation program disclosed in Patent Document 1, it is difficult to estimate the food and calories from the captured image, and there is a problem that it may not be possible to support the monitoring or health management of the person to be monitored.

[0006] The present disclosure has been made to solve the above problems, and an object thereof is to obtain a meal monitoring device that can support the monitoring or health management of a person to be monitored without estimating the food or calories.

Means for Solving the Problems

[0007] The meal monitoring device of the present disclosure includes a tableware image obtained by extracting, from tableware information including a tableware area, which is an area of tableware detected from an image of a table of a person to be monitored, and information regarding the position of the tableware on the table, a meal situation estimation unit that estimates the meal situation of the person to be monitored, and a situation requiring attention determination unit that determines whether the situation of the person to be monitored corresponds to a situation requiring attention by using the meal situation and model data indicating a meal situation determined to be normal.

Advantages of the Invention

[0008] Therefore, the meal monitoring device of the present disclosure can estimate the meal situation of the person to be monitored from the tableware information created from the acquired image of the table, compare it with the model data indicating the meal situation determined to be normal, and determine whether the situation of the person to be monitored corresponds to a situation requiring attention. Accordingly, the meal monitoring device of the present disclosure can assist in monitoring or health management of the person to be monitored without estimating the dishes or calories.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described with reference to the drawings. The same reference numerals are given to common or corresponding elements in each figure to simplify or omit the description. Also, the configurations shown in the embodiments below are examples of the technical idea according to the present disclosure, and it is possible to combine them with other known technologies, or to combine a plurality of technical ideas described in the present disclosure. Further, it is possible to omit or change a part of the configuration without departing from the gist of the present disclosure.

[0011] Embodiment 1. FIG. 1 is a configuration diagram of the meal monitoring system 1 in Embodiment 1. Also, FIG. 2 is a functional block diagram of the meal monitoring system in Embodiment 1. The meal monitoring system 1 is a system that supports the monitoring or health management of a person to be monitored from the meal situation. The person to be monitored is, for example, a single person or an elderly person. The meal monitoring system 1 includes an image acquisition device 11, a processing device 12, a server device 13, and a display device 14. The image acquisition device 11, the processing device 12, the server device 13, and the display device 14 that make up the meal monitoring system 1 transmit and receive information to and from each other by wire or wirelessly.

[0012] The image acquisition device 11 is a device that acquires an image of the table at which the person to be monitored eats. The image acquisition device 11 is installed at a position where it can photograph the upper surface of the table from above the table. Also, the image acquisition device 11 may be installed at a position where it can photograph the table obliquely as long as it can acquire an image capable of performing a process of detecting a utensil region described later. The image acquisition device 11 repeatedly acquires images every time a certain period of time elapses. Here, the certain period of time may be, for example, 1 minute, 5 minutes, or 10 minutes. The image acquisition device 11 is realized by, for example, a camera, and photographs the table to acquire an image. The image acquisition device 11 includes an image acquisition unit 111. The image acquisition unit 111 is configured to have a function of acquiring an image of the table of the person to be monitored. The image acquisition unit 111 acquires an image of the entire table and outputs the acquired image to the processing device 12.

[0013] The processing device 12 is a device that performs information processing on the image acquired by the image acquisition device 11 and extracts utensil information. Note that the utensil information will be described later. The processing device 12 includes a utensil region detection unit 121 and a utensil information creation unit 122. The processing device 12 is installed, for example, in the house of the person to be monitored.

[0014] The tableware area detection unit 121 detects, for each piece of tableware, the tableware area which is the area occupied by the tableware from the image of the dining table acquired by the image acquisition unit 111. The tableware area is detected, for example, by image recognition or pattern matching. Here, the tableware means the tableware used when the person being monitored eats at the dining table. FIG. 3 is a schematic diagram showing the tableware area C detected by the tableware area detection unit 121 from the image A of the dining table. In FIG. 3, an example is shown where the entire dining table B is reflected in the image A of the dining table. The tableware area C which is the area occupied by the tableware D is defined, for example, as the area surrounded by the dotted rectangle including the tableware D inside as shown in FIG. 3. Note that if the tableware area C is defined as the area surrounded by a rectangle, for example, even if medicine or the like is shown in the image A of the dining table, only the image of the area defined by the rectangle is transmitted to the server device 13, so that the part where the medicine is shown can be excluded and personal information can be protected. Also, even when a document with personal information such as an address or name is shown in the image of the dining table, personal information can be protected. Note that the definition of the tableware area C is not limited to a rectangle, and it may be defined as an area along the contour of the tableware.

[0015] The tableware information creation unit 122 creates tableware information. Here, the tableware information is information including the tableware image obtained by extracting the tableware area detected by the tableware area detection unit 121 from the image of the dining table and information indicating the position of the tableware on the dining table. The tableware image is obtained, for example, by cutting out the rectangular area surrounded by the dotted line shown in FIG. 3 from the image.

[0016] The information indicating the position of the tableware on the dining table is, for example, the information indicating the position of the center of the detected tableware area by the coordinates represented in an XY coordinate system (not shown). The XY coordinate system is, for example, a coordinate system with the lower left corner of the image of the dining table as the origin.

[0017] FIG. 4 is a schematic diagram showing an example of the tableware information transmitted from the tableware information creation unit 122 to the tableware information acquisition unit 131. As shown in FIG. 4, the tableware information is information in which, for example, numbers and information indicating the position of the tableware on the table are associated with the tableware images extracted by the tableware information creation unit 122. The tableware information includes, for example, the date and time when the image acquisition unit 111 acquired the image of the table. The number may be any number or symbol, as long as it can distinguish the detected tableware.

[0018] The server device is a device that estimates the meal situation from the tableware information extracted by the processing device 12, determines whether the person to be monitored for meals is in a situation that requires attention based on the estimated meal situation, and notifies the display device 14. Note that the meal situation and the situation requiring attention will be described later. The server device 13 includes a tableware information acquisition unit 131, a tableware information storage unit 132, a meal situation estimation unit 133, a storage unit 134, a situation requiring attention determination unit 135, and a notification unit 136.

[0019] The tableware information acquisition unit 131 acquires the tableware information created by the tableware information creation unit 122.

[0020] The tableware information storage unit 132 stores the tableware information acquired by the tableware information acquisition unit 131.

[0021] The meal situation estimation unit 133 estimates the meal situation of the person to be monitored at the date and time included in the tableware information from the tableware information stored in the tableware information storage unit 132. Further, the meal situation estimation unit 133 may estimate, for example, at least one of the color, shape, size, or number of the tableware from the tableware information and use that information to estimate the meal situation. Note that the meal situation estimation unit 133 may perform the estimation using two or more pieces of information. In this case, the meal situation estimation unit 133 can improve the accuracy of the meal situation estimation compared to the case where one piece of tableware information is used.

[0022] The meal situation refers to a situation that can be estimated from, for example, the number of diners, the number of dishes, the seats used, the tableware used, the length of the meal time estimated from whether it is during a meal, or the amount of food eaten, etc., which can be estimated from the state of the tableware placed on the table, and means the situation when the person under surveillance has a meal. Note that the meal situation is estimated based on the tableware information created from the image of the table taken during the meal of the person under surveillance. Also, the meal situation estimation unit 133 may estimate the meal situation using the tableware information created from the image of the table taken at a certain time, or may estimate the meal situation using a plurality of pieces of tableware information created from images of a plurality of tables taken at different times. The number of pieces of tableware information used for estimating the meal situation varies depending on the type of meal situation to be estimated.

[0023] Here, a method for estimating the color, shape, size, and number of tableware used for estimating the meal situation will be described. The color of the tableware is, for example, information indicating the color of the area corresponding to the tableware extracted from the tableware image as a numerical value. The numerical value is, for example, an RGB value. In this case, the color of the tableware can be quantitatively shown.

[0024] The shape of the tableware is, for example, information indicating the ratio of the vertical length to the horizontal length of the tableware area. In this case, the shape of the tableware can be quantitatively shown.

[0025] The size of the tableware is, for example, information indicated by a value calculated based on the vertical length and the horizontal length of the tableware area in an XY coordinate system (not shown). The vertical length and the horizontal length of the tableware area are, for example, the vertical length and the horizontal length of a defined rectangular area. The vertical length of the rectangle is calculated, for example, from the distance between the coordinates of the lower left vertex and the upper left vertex of the rectangle in a coordinate system with the lower left corner of the table image as the origin. The horizontal length of the rectangle is calculated, for example, from the distance between the coordinates of the lower left vertex and the lower right vertex of the rectangle in a coordinate system with the lower left corner of the table image as the origin.

[0026] The number of tableware is, for example, information indicating the total number of tableware areas detected from an image of one table.

[0027] Next, a method for estimating the eating situation will be described. FIG. 5 is a diagram showing an example of the eating situation estimated by the eating situation estimation unit 133 and the estimation method. As shown in FIG. 5, the eating situation estimation unit 133 groups a plurality of utensils photographed at the same time using the utensil information. For example, the eating situation estimation unit 133 extracts the same menu group, which is a group of utensils in which at least one of the size, shape, and color matches among the utensils photographed at the same time. The eating situation estimation unit 133 estimates the number of utensils included in the same menu group as the number of people eating at the same time when the person to be monitored eats. In this case, for example, when a person eating temporarily leaves the table, by detecting whether it is a temporary departure from the movement situation of the person eating and comparing it with the case of judging the number of people eating at the same time, the number of people eating at the same time can be judged more simply.

[0028] Also, the eating situation estimation unit 133 may estimate the number of the same menu groups as the number of items. In this case, the eating situation estimation unit 133 can estimate the number of items more accurately than estimating the dishes.

[0029] In addition, the meal situation estimation unit 133 may refer to information indicating the positions of the seats registered in advance, and estimate the seats used during the meal from the information indicating the positions of the utensils. In this case, if the user of the seat is determined in advance for each seat to be used, among the persons to be monitored, it may be possible to identify who is the person who had the meal. Here, the information indicating the positions of the seats registered in advance is, for example, information in which coordinates indicating the positions of the seats in the same XY coordinate system as the XY coordinates used for the information indicating the positions of the utensils are assigned to each seat of the dining table in advance. The information indicating the positions of the utensils is also information indicated by coordinates. Therefore, the meal situation estimation unit 133 may calculate and compare the distances between the utensils and the respective seats from the information indicating the positions of the utensils and the information indicating the positions of the seats, and estimate that the seat with the closest distance was used. When the seat with the closest distance varies depending on the utensil, the meal situation estimation unit 133 may estimate that a plurality of seats were used, and estimate the number of seats used as the number of persons having the meal simultaneously. In this case, for example, even if a person having the meal temporarily leaves the seat, the number of persons who had the meal can be determined more simply as compared with the case of detecting the movement of the person having the meal.

[0030] In addition, when the person using the utensil is determined, the meal situation estimation unit 133 may estimate the person who had the meal by using the correspondence relationship between the utensils and the persons registered in advance. In this case, since the estimation is performed based on the daily information of the persons to be monitored, the meal situation estimation unit 133 can accurately estimate the person who had the meal.

[0031] In addition, the meal situation estimation unit 133 may estimate the types of utensils used from the color, shape, and size of the utensils. For example, the meal situation estimation unit 133 may register in advance the color, shape, and size of the utensils that the person to be monitored may use for each type of utensil, and estimate the type of utensil used by comparing the color, shape, or size of the utensil registered in advance with the color, shape, or size of the utensil extracted from the image of the dining table.

[0032] Also, when the amount of food intake is determined by the tableware, the types of tableware and the numerical values indicating the amount of food intake may be registered in advance in correspondence, and the meal situation estimation unit 133 may estimate the amount of food intake from the type of tableware. For example, a case will be described where the numerical value indicating the amount of food intake corresponding to a bowl is set to 3, the numerical value indicating the amount of food intake corresponding to a rice bowl is set to 1, and the numerical value indicating the amount of food intake corresponding to a small bowl is set to 0.5. The total of the numerical values indicating the amount of food intake when only one bowl is used is 3, whereas the total of the numerical values indicating the amount of food intake when one rice bowl and two small bowls are used is 2. Thus, the amount of food intake in cases where the number or type of tableware is different can be indicated numerically. Further, since the meal situation estimation unit 133 estimates the amount of food intake from the number of tableware, the shape of the tableware, and the color of the tableware, the normal amount of food intake of the person under observation can be calculated in a simpler and more accurate manner than when calculating calories from the amount of food intake.

[0033] Also, the meal situation estimation unit 133 may estimate the number of tableware from the tableware information, estimate whether the person is in the middle of a meal, and estimate the length of the meal time. FIG. 6 is a schematic diagram showing a method by which the meal situation estimation unit 133 estimates the meal time from the number of tableware. As shown in FIG. 6, the time from time T1 to time T2 when the number of tableware placed on the dining table is increasing is the time when the person under observation is preparing the meal, the time from time T2 to time T3 when the number of tableware placed on the dining table is at its maximum and constant is the time when the person under observation is eating, and the time from time T3 to time T4 when the number of tableware placed on the dining table is decreasing is the time when the person under observation is cleaning up. The meal situation estimation unit 133 estimates the length of the meal time. Note that the start time and end time of the meal may be estimated and recorded from time T2 when the number of tableware stops increasing and time T3 when the number of tableware starts decreasing, and the length of the meal time may be estimated from the difference between the start time and the end time. In these cases, the length of the meal time can be estimated simply.

[0034] The meal situation estimated by the meal situation estimation unit 133 is stored in the storage unit 134. FIG. 7 is a table showing an example of the meal situation stored in the storage unit 134. As shown in FIG. 7, the storage unit 134 stores, for example, information indicating the start of a meal, the end of a meal, the number of items, the number of people eating at the same time, the seats used, the utensils used, or the amount of food eaten, together with the date and time when the image of the dining table used for estimation was acquired. Note that the seats A, C, and D shown in FIG. 7 mean the seats used among the seats numbered A to F assigned to each seat in advance when a six-seat dining table is used. The meal situation is stored in the storage unit 134 in the order in which the meal situation estimation unit 133 can estimate it. For example, when the image is unclear, when the utensils are not detected, when the number of people eating is unknown, or when the end of the meal cannot be estimated even though the start of the meal can be estimated, the meal situation is not stored in the storage unit 134.

[0035] Note that when the meal situation estimated by the meal situation estimation unit 133 is the number of meals, since the number of meals is information obtained daily, it may be stored in the storage unit 134 together with only the month and day instead of the date and time, and the time does not necessarily have to be stored. Also, when estimating the meal situation using a plurality of utensil information, that is, images of a plurality of dining tables, such as the meal time, it is also possible to store the meal situation together with information corresponding to the meal situation instead of the date and time. For example, when storing the meal time, the meal situation is stored together with the date and the estimated length of the meal time.

[0036] Note that the attention-required situation determination unit 135 determines whether the situation corresponds to an attention-required situation based on the change in the meal situation stored in the storage unit 134. The attention-required situation means, for example, when comparing the meal situation of the person under watch with the model data, irregularity in meal times, changes in the length of meal times, decrease in the number of items, decrease in the number of people having meals, decrease in the number of times of using utensils, or increase or decrease in the amount of food, etc. It means a situation where the change in the life of the person under watch is large and it is expected to be a situation different from the normal situation, based on the magnitude of the deviation from the model data in the meal situation of the person under watch. Note that the model data is data indicating a meal situation determined to be normal, and it may be data on the meal situation of the person under watch in the past normal times, or it may be data on an arbitrary meal situation set in advance. Thus, based on the change in the meal situation including the current meal situation of the person under watch, the attention-required situation determination unit 135 determines whether it corresponds to an attention-required situation, so that the state of the person under watch can be appropriately determined. Also, when using the data on the meal situation of the person under watch in the past normal times as the model data, since the attention-required situation determination unit 135 makes a determination using the data corresponding to each person under watch, an accurate determination can be made for each person under watch. Also, when using the data on an arbitrary meal situation set in advance as the model data, the meal watching system 1 can be used immediately as compared with the case of collecting the data on the past meal situation of the person under watch.

[0037] FIG. 8 is a table showing an example of a method by which the attention-required situation determination unit 135 determines whether the situation corresponds to an attention-required situation based on the change in the meal situation. For example, when the frequency of the start time of a meal deviating from the model data by a predetermined value or more increases, the attention-required situation determination unit 135 determines that the meal time is irregular and corresponds to an attention-required situation.

[0038] Also, for example, when the frequency of the length of the meal time deviating from the model data by a predetermined value or more increases, the attention-required situation determination unit 135 determines that the length of the meal time has changed and corresponds to an attention-required situation.

[0039] Further, for example, when the frequency of the quantity of items decreasing to less than a predetermined value from the model data increases, the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation, assuming that the quantity of items has decreased from before.

[0040] Further, for example, when the frequency of the number of people eating together being less compared to the number of people under surveillance registered in advance increases, the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation, assuming that the people under surveillance are not eating together. In this case, the model data is the number of people under surveillance registered in advance, and the people under surveillance are people who are expected to usually eat together in multiple people, such as elderly couples.

[0041] When a seat or a person who has eaten is specified, when there are two or more people under surveillance, the attention-required situation determination unit 135 can determine in the following-described process who has changed their eating situation and who corresponds to the attention-required situation.

[0042] Further, for example, when the frequency of the number of uses of a specific utensil decreasing to less than a predetermined value from the model data increases, the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation, assuming that what was eaten in the past is no longer eaten. The specific utensil is a utensil that is registered in advance as a utensil used for serving dishes that the people under surveillance frequently eat.

[0043] Further, for example, when the frequency of the total amount per meal of the numerical value indicating the amount of food deviating from a predetermined value by more than a predetermined value from the model data increases, the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation, assuming that the amount of food has changed.

[0044] When the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation, the notification unit 136 transmits the notification content to the display device 14. The notification content may include information specifically indicating the corresponding attention-required situation.

[0045] The display device 14 is installed, for example, at the home of a child living separately from the person under surveillance or at the base of a nursing service. In this case, when the attention-required situation determination unit 135 transmits the notification content to the contact information of the child's home or the nursing service base registered in advance, it becomes possible to notify the child or the person in charge of the nursing service that the situation corresponds to an attention-required situation. The display device 14 is realized by, for example, a mobile terminal, a tablet, or a display. The display device 14 includes a display unit 141. The display unit 141 displays that the person under surveillance corresponds to an attention-required situation. Further, the display unit 141 may display the specific attention-required situation that corresponds. The display unit 141 is realized by, for example, the display screen of a mobile terminal, the display screen of a tablet, or the display screen of a display.

[0046] Next, the hardware configuration of the server device 13 will be described. FIG. 9 is a hardware configuration diagram showing the hardware configuration of the server device 13. The server device 13 is composed of an arithmetic device 2, a storage device 3, an input device 4, an auxiliary storage device 5, an output device 6, and a signal line 7.

[0047] The processing performed by the server device 13 can be realized by, for example, the arithmetic device 2. The arithmetic device 2 realizes the functions of the server device 13 by reading the necessary programs from the auxiliary storage device 5 and executing the processing. The arithmetic device 2 is, for example, a processor, and the processor is an IC (Integrated Circuit) that performs arithmetic processing. Specific examples of the processor are a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). Further, it may be a personal computer, a microcomputer board, or an FPGA (Field Programmable Gate Array) board.

[0048] The storage device 3 is the main storage device of the server device 13. The main storage device temporarily stores the results of the calculation processing performed by the arithmetic device 2. For example, the storage device 3 is a RAM (Random Access Memory).

[0049] The input device 4 inputs input data such as the device information created by the device information creation unit 122 to the server device 13. The input device 4 is an input interface of the server device 13.

[0050] The auxiliary storage device 5 has the function of the storage unit 134 and is an auxiliary storage device of the server device 13. For example, the auxiliary storage device 5 stores programs necessary for the operation of the server device 13, the device information created by the device information creation unit 122, the meal situation estimated by the meal situation estimation unit 133, and the like. For example, the auxiliary storage device 5 is a ROM (Read Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

[0051] The output device 6 outputs output data to the display device 14. The output device 6 is an output interface of the server device 13.

[0052] The signal line 7 is a transmission path for transmitting and receiving data between the respective components described in FIG. 9.

[0053] Note that the hardware configuration of the processing device 12 is also composed of an arithmetic device, a storage device, an input device, an auxiliary storage device, an output device, and a signal line, similar to the server device 13. The processing performed by the processing device 12 can be realized by, for example, an arithmetic device. The description of the other components will be omitted.

[0054] Next, the operation of the meal monitoring system 1 will be described. FIG. 10 is a flowchart showing the operation of the meal monitoring system 1. When the power of the devices constituting the meal monitoring system 1 is turned on, the meal monitoring system 1 starts operating. Note that the start of the operation may also be performed by setting a dedicated application of the meal monitoring system 1. In step S1, the image acquisition unit 111 acquires an image of the dining table. When the image acquisition unit 111 acquires an image of the dining table, the process proceeds to step S2.

[0055] In step S2, the tableware area detection unit 121 detects the tableware area from the image of the dining table acquired by the image acquisition unit 111. When the tableware area detection unit 121 detects the tableware area from the image, the process proceeds to step S3. In step S3, the tableware information creation unit 122 creates tableware information including the tableware image obtained by extracting the tableware area from the image of the dining table and information regarding the position of the tableware on the dining table. The tableware information creation unit 122 transmits the tableware information to the tableware information acquisition unit 131. When the tableware information creation unit 122 transmits the tableware information, the process proceeds to step S4.

[0056] In step S4, the tableware information storage unit 132 stores the tableware information acquired by the tableware information acquisition unit 131. When the tableware information storage unit 132 stores the tableware information, the process proceeds to step S5.

[0057] In step S5, the meal situation estimation unit 133 estimates the meal situation of the person under surveillance from the tableware information. When the meal situation estimation unit 133 estimates the meal situation of the person under surveillance, the process proceeds to step S6.

[0058] In step S6, the storage unit 134 stores the meal situation estimated by the meal situation estimation unit 133. When the storage unit 134 stores the meal situation, the process proceeds to step S7.

[0059] In step S7, the situation requiring attention determination unit 135 determines whether the meal situation of the person under surveillance corresponds to a situation requiring attention based on the change in the meal situation. If the meal situation of the person under surveillance does not correspond to a situation requiring attention, the process proceeds to step S10. If the meal situation of the person under surveillance corresponds to a situation requiring attention, the process proceeds to step S8.

[0060] In step S8, the notification unit 136 notifies a pre-registered contact that the meal situation of the person under surveillance corresponds to a situation requiring attention. When the notification unit 136 notifies the pre-registered contact, the process proceeds to step S9.

[0061] In step S9, the meal monitoring system 1 determines whether to end the process. If the process is to be ended, the process ends. The end of the meal monitoring system 1 may be, for example, ended by the app setting or ended by turning off the power. However, regardless of step S9, if the power of the device constituting the meal monitoring system 1 is turned off, the operation ends at that time. If the process is not ended, the process proceeds to step S10.

[0062] In step S10, the image acquisition unit 111 waits until a certain period of time has elapsed since the latest image was acquired. Here, the certain period of time is, for example, the time interval until the time required for a series of processes from when the image acquisition unit 111 acquires an image until the notification unit 136 makes a notification has elapsed. When a certain period of time has elapsed since the image acquisition unit 111 acquired the latest image, the process returns to step S1, and the image acquisition unit 111 acquires the next image.

[0063] Note that the meal monitoring system 1 may not perform the operation in one flow, but may be divided into two processes and executed independently. Specifically, steps S1 to S4, which are the processes from when the image acquisition unit 111 acquires an image until the utensil information storage unit 132 stores the utensil information, and steps S5 to S9, which are the processes from when the meal situation estimation unit 133 estimates the meal situation until the attention required situation determination unit 135 determines whether it corresponds to an attention required situation, and if it corresponds to an attention required situation, notifies the contact, may be executed independently by the meal monitoring system 1.

[0064] Note that in this embodiment, an example in which the meal situation estimation unit 133 identifies the person who ate from the seat or utensil used has been described. However, when a person appears in the image acquired by the image acquisition unit 111, the person may be identified from the image. The identification of the person is performed, for example, by image recognition.

[0065] In the present embodiment, the case where the processing device 12 and the image acquisition device 11 are separate devices has been described. However, for example, the processing device 12 may be provided in the camera which is the image acquisition device 11.

[0066] Also, the case where the processing device 12 and the server device 13 are separate devices has been described. However, for example, the processing device 12 may be provided in the server device 13. In this case, the image of the dining table acquired by the image acquisition device 11 will be directly transmitted to the server device 13.

[0067] Conventionally, for dishes, even if the dishes have different ingredients and nutrients, they may sometimes look almost the same. Also, even for the same dish, the appearance may vary greatly from household to household. Therefore, it is difficult to estimate the calories disclosed in Patent Document 1. However, according to the configuration of the meal monitoring system 1 of Embodiment 1, from the tableware information created from the acquired image of the dining table, a meal situation estimation unit that estimates the meal situation of the person to be monitored, and a model data indicating the meal situation and the meal situation determined to be normal are used. And a situation requiring attention determination unit that determines whether the situation of the person to be monitored corresponds to a situation requiring attention. For this reason, instead of estimating the dish or calories from the image, the meal monitoring system of the present disclosure estimates the meal situation of the person to be monitored from the tableware information created from the acquired image, and uses the meal situation and the model data indicating the meal situation determined to be normal. It is possible to determine whether the situation of the person to be monitored corresponds to a situation requiring attention. That is, the meal monitoring system of the present disclosure estimates the meal situation using the tableware information that can be accurately created from the image as compared with the calorie estimation by estimating the dish, and determines whether the situation of the person to be monitored corresponds to a situation requiring attention using the estimated meal situation. Therefore, the meal monitoring system of the present disclosure can assist in monitoring or health management of the person to be monitored without estimating the dish or calories.

[0068] Embodiment 2. In Embodiment 1, the meal situation estimation unit 133 described a method of estimating the meal situation based on rule-based processing. In Embodiment 2, an example in which the meal situation estimation unit 133 estimates the meal situation using AI (Artificial Intelligence) learning will be described. Hereinafter, as an example of estimating the meal situation, a case of estimating the number of meals per day will be described. Note that descriptions of the same parts as in Embodiment 1 will be omitted.

[0069] <Learning phase> FIG. 11 is a configuration diagram of the learning device 15 for the meal situation estimation unit 133. The learning device 15 includes a data acquisition unit 151, a model generation unit 152, and a learned model storage unit 153.

[0070] The data acquisition unit 151 acquires, as learning data, a combination of the tableware information of the person under surveillance for one day and the data on the number of meals on the day corresponding to the tableware information for one day. The day corresponding to the tableware information for one day is the day indicated by the date and time included in the tableware information for one day, and is the day on which the image of the dining table used for extracting the tableware information was acquired.

[0071] The model generation unit 152 learns the number of meals per day based on the learning data created based on the combination of the tableware information of the person under surveillance for one day output from the data acquisition unit 151 and the data on the number of meals, which is the actual number of times the person under surveillance ate at the dining table on the day corresponding to the tableware information for one day. That is, the model generation unit 152 generates a learned model for inferring the number of meals per day from the tableware information of the person under surveillance for one day stored in the tableware information storage unit 132. Here, the learning data is data that associates the tableware information of the person under surveillance for one day and the number of meals on the day corresponding to the tableware information for one day with each other. Since the state of the tableware arranged on the dining table changes depending on the situation such as not eating, preparing for a meal, eating, or cleaning up after a meal, the number of meals on the day corresponding to the tableware information for one day can be inferred from the tableware information for one day.

[0072] Note that although the learning device 15 has been described in the case where it is used to learn the number of meals in the meal situation estimation unit 133, the learning device 15 may be a device different from the meal situation estimation unit 133. For example, the learning device 15 may be connected to the meal situation estimation unit 133 via a network. Further, the learning device 15 may be built into the meal situation estimation unit 133. Furthermore, the learning device 15 and the inference device 16 may exist on a cloud server.

[0073] As the learning algorithm used by the model generation unit 152, known algorithms such as supervised learning, unsupervised learning, and reinforcement learning can be used. As an example, the case of applying a neural network will be described.

[0074] The model generation unit 152 learns the number of meals for one day, for example, by so-called supervised learning according to a neural network model. Here, supervised learning refers to a method of giving a set of input and result (label) data to the learning device 15, learning the features in the learning data, and inferring the result from the input.

[0075] A neural network is composed of an input layer consisting of a plurality of neurons, an intermediate layer (hidden layer) consisting of a plurality of neurons, and an output layer consisting of a plurality of neurons. The intermediate layer may be one layer or two or more layers.

[0076] FIG. 12 is a schematic diagram showing an example of a three-layer neural network. For example, in the case of a three-layer neural network as shown in FIG. 12, when a plurality of inputs are input to the input layer (X1 to X3), the values are multiplied by weights W1 (w11 to w16) and input to the intermediate layer (Y1 to Y2), and the result is further multiplied by weights W2 (w21 to w26) and output from the output layer (Z1 to Z3). This output result changes depending on the values of the weights W1 and W2.

[0077] In the present application, the neural network learns according to the learning data created based on the combination of the daily instrument information of the person under surveillance acquired by the data acquisition unit 151 and the number of meals on the day corresponding to the daily instrument information, so that the number of meals output when the daily instrument information is input approaches the correct number of meals through so-called supervised learning.

[0078] That is, the neural network learns by adjusting the weights W1 and W2 so that the result output from the output layer after inputting the daily instrument information of the person under surveillance into the input layer approaches the number of meals on the day corresponding to the daily instrument information.

[0079] The model generation unit 152 generates and outputs a learned model by executing the above learning.

[0080] The learned model storage unit 153 stores the learned model output from the model generation unit 152.

[0081] Next, with reference to FIG. 13, the process by which the learning device learns will be described. FIG. 13 is a flowchart related to the learning process of the learning device.

[0082] In step S41, the data acquisition unit 151 acquires the combination of the daily instrument information of the person under surveillance and the number of meals on the day corresponding to the daily instrument information. Although it is assumed that the combination of the daily instrument information of the person under surveillance and the number of meals on the day corresponding to the daily instrument information is acquired simultaneously, it is only necessary to be able to input the daily instrument information of the person under surveillance and the number of meals on the day corresponding to the daily instrument information in association with each other, and the daily instrument information of the person under surveillance and the data on the number of meals on the day corresponding to the daily instrument information may be acquired at different timings.

[0083] In step S42, the model generation unit 152 learns the number of meals per day by so-called supervised learning according to the learning data created based on the combination of the information of the utensils used by the person under care for one day acquired by the data acquisition unit 151 and the number of meals on the day corresponding to the information of the utensils used for one day, and generates a learned model.

[0084] In step S43, the learned model storage unit 153 stores the learned model generated by the model generation unit 152.

[0085] <Utilization phase> FIG. 14 is a configuration diagram of the inference device 16 related to the meal situation estimation unit 133. The inference device 16 includes a data acquisition unit 161 and an inference unit 162.

[0086] The data acquisition unit 161 acquires the information of the utensils used by the person under care for one day from the utensil information storage unit 123.

[0087] The inference unit 162 infers the number of meals per day using the learned model stored in the learned model storage unit 153. That is, by inputting the information of the utensils used by the person under care for one day acquired by the data acquisition unit 161 into this learned model, it is possible to output the number of meals per day inferred from the information of the utensils used by the person under care for one day.

[0088] In this embodiment, the description has been made on the assumption that the number of meals per day is inferred using the learned model learned by the model generation unit 152 of the meal situation estimation unit 133. However, a learned model may be acquired from outside, such as another meal situation estimation unit 133, and the number of meals per day may be inferred based on this learned model.

[0089] Next, with reference to FIG. 15, the process for obtaining the number of meals per day using the inference device 16 will be described. FIG. 15 is a flowchart showing the process in which the inference device 16 performs an inference process.

[0090] In step S44, the data acquisition unit 161 acquires the information of the utensils used by the person under care for one day.

[0091] In step S45, the inference unit 162 acquires the learned model from the learned model storage unit 153, inputs the utensil information of the person to be monitored for one day into the acquired learned model, and obtains the number of meals for one day.

[0092] In step S46, the inference unit 162 outputs the number of meals for one day obtained by the learned model to the storage unit 134.

[0093] The attention-required situation determination unit 135 determines whether it corresponds to an attention-required situation from the number of meals for one day stored in the storage unit 134. For example, when the frequency of deviation of the number of meals for one day from a predetermined value in the model data increases, the attention-required situation determination unit 135 determines that it corresponds to an attention-required situation on the grounds that the number of meals is irregular. Regarding the operations after it is determined that it corresponds to an attention-required situation and after it is determined that it does not correspond to an attention-required situation, they are the same as those in the first embodiment.

[0094] In this embodiment, the case where supervised learning is applied to the learning algorithm used by the model generation unit 152 has been described, but it is not limited thereto. Regarding the learning algorithm, in addition to supervised learning, reinforcement learning, unsupervised learning, semi-supervised learning, etc. can also be applied.

[0095] Also, as the learning algorithm used for the model generation unit 152, deep learning, which learns the extraction of the feature quantity itself, can be used, and machine learning may be executed according to other known methods, such as genetic programming, inductive logic programming, and support vector machines.

[0096] In addition, the learning device 15 in the present embodiment generates a learned model that outputs the number of meals per day from the combination of the daily appliance information and the data of the number of meals per day corresponding to the daily appliance information. The inference device 16 has been described as an example of outputting the number of meals per day using the learned model. However, the learning device 15 inputs the daily data of the days when the number of meals per day does not exceed the threshold value and the daily data of the days when the threshold value is exceeded as learning data. When the daily appliance information of a certain day is input, a learned model that outputs whether the number of meals on that day exceeds the threshold value or not is generated. The inference device 16 may output whether the number of meals per day exceeds the threshold value or not using the learned model. In this case, when the daily appliance information is input to the learned model, it will be output whether it corresponds to a situation that requires attention.

[0097] In addition, in the second embodiment, as an example of estimating the meal situation, the case of estimating the number of meals per day has been described. However, by changing the input to the learning data and the learned model according to the meal situation to be estimated, other meal situations can be estimated in the same way. For example, when estimating the number of diners, the learning data may be created from the combination of the appliance information at a certain time and the number of diners who were actually having a meal together when the image of the dining table used to extract the appliance information was acquired.

[0098] According to the present embodiment, since the meal monitoring system 1 includes the learning device 15 that generates a learned model for inferring the meal situation from the appliance information using the learning data created from the combination of the appliance information and the meal situation when the image of the dining table from which the appliance information was extracted was acquired, a learned model for inferring the meal situation of the person under monitoring when the image of the dining table from which the appliance information of the person under monitoring was extracted is acquired can be obtained.

[0099] In addition, since the meal monitoring system 1 includes an inference device 16 that infers the meal situation when acquiring an image of a dining table from which utensil information has been extracted using a learned model, it is possible to obtain the meal situation of the person being monitored when acquiring an image of the dining table from which utensil information has been extracted from the utensil information of the person being monitored.

[0100] Hereinafter, various aspects of the present disclosure will be summarized and described as appendices.

[0101] (Appendix 1) A meal situation estimation unit that estimates the meal situation of the person being monitored from utensil information including a utensil image obtained by extracting from the image a utensil region that is a region of a utensil detected from an image of the dining table of the person being monitored and information regarding the position of the utensil on the dining table; A situation requiring attention determination unit that determines whether the situation of the person being monitored corresponds to a situation requiring attention using the meal situation and model data indicating a meal situation determined to be normal; A meal monitoring device. (Appendix 2) The meal situation estimation unit estimates the meal situation using at least one of the color, shape, size, or number of the utensils estimated from the utensil information. The meal monitoring device according to Appendix 1. (Appendix 3) The meal situation is a situation when the person being monitored has a meal, indicated by at least one of the number of people having a meal, the number of items, the seats used, the person who had the meal, the utensils used, the length of the meal time, or the amount of food eaten. The meal monitoring device according to Appendix 1 or Appendix 2. (Appendix 4) The situation requiring attention is a situation including at least one of irregularity of the meal time, change in the length of the meal time, decrease in the number of items, decrease in the number of people having a meal, decrease in the number of times the utensils are used, or increase or decrease in the amount of food eaten. The meal monitoring device according to Appendices 1 to 3. (Appendix 5) An image acquisition unit that acquires an image of the dining table of the person being monitored; A utensil region detection unit that detects a utensil region that is a region of a utensil from the image; An appliance information creation unit that creates appliance information including an appliance image obtained by extracting the detected appliance area from the image and information regarding the position of the appliance on the table; A meal situation estimation unit that estimates the meal situation of the person under surveillance from the appliance information; A situation requiring attention determination unit that compares the meal situation with model data indicating a normal meal situation and determines whether the situation of the person under surveillance corresponds to a situation requiring attention; A notification unit that, when the situation requiring attention determination unit determines that the situation corresponds to a situation requiring attention, notifies a pre-registered notification destination A meal monitoring system comprising the above. (Appendix 6) An image acquisition step of acquiring an image of the table of the person under surveillance; An appliance area detection step of detecting an appliance area that is an area of an appliance from the image; An appliance information creation step of creating appliance information including an appliance image obtained by extracting the detected appliance area from the image and information regarding the position of the appliance on the table; A meal situation estimation step of estimating the meal situation of the person under surveillance from the appliance information; A situation requiring attention determination step of comparing the meal situation with model data indicating a normal meal situation and determining whether the situation of the person under surveillance corresponds to a situation requiring attention; A meal monitoring method comprising the above. (Appendix 7) The meal monitoring method according to Appendix 6, further comprising a notification step of, when it is determined in the situation requiring attention determination step that the situation of the person under surveillance corresponds to a situation requiring attention, notifying a pre-registered notification destination The meal monitoring method according to Appendix 6. (Appendix 8) A meal situation estimation step of estimating the meal situation of the person under surveillance from appliance information including an appliance image obtained by extracting from the image an appliance area that is an area of an appliance detected from an image of the table of the person under surveillance and information regarding the position of the appliance on the table; A step of determining whether the situation of the person under surveillance corresponds to a situation requiring attention by comparing the dietary situation with model data indicating a dietary situation determined to be normal, A dietary surveillance program for causing a computer to execute. (Appendix 9) In the step of determining the situation requiring attention, when it is determined that the situation of the person under surveillance corresponds to a situation requiring attention, it further includes a notification step of notifying a pre-registered notification destination The dietary surveillance program according to Appendix 8. (Appendix 10) A data acquisition unit that acquires learning data including a utensil image obtained by extracting from the image a utensil area that is an area of a utensil detected from an image of the dining table of the person under surveillance, and utensil information including information on the position of the utensil on the dining table, and the dietary situation when the image was acquired; A model generation unit that generates a learned model for inferring the dietary situation from the utensil information using the learning data; A learning device comprising: (Appendix 11) A data acquisition unit that acquires utensil information including a utensil image obtained by extracting from the image a utensil area that is an area of a utensil detected from an image of the dining table of the person under surveillance, and information on the position of the utensil on the dining table; An inference unit that outputs the dietary situation of the person under surveillance from the utensil information of the person under surveillance acquired by the data acquisition unit using a learned model for inferring the dietary situation from the utensil information; An inference device comprising:

Explanation of Signs

[0102] 1 Diet monitoring system, 2 Arithmetic unit, 3 Memory device, 4 Input device, 5 Auxiliary storage device, 6 Output device, 7 Signal line, 11 Image acquisition device, 12 Processing unit, 13 Server device, 14 Display device, 15 Learning device, 16 Inference device, 111 Image acquisition section, 121 Utensil area detection section, 122 Utensil information creation section, 131 Utensil information acquisition section, 132 Utensil information storage section, 133 Meal situation estimation section, 134 Storage section, 135 Attention required situation determination section, 136 Notification section, 141 Display section, 151 Data acquisition section, 152 Model generation section, 153 Trained model storage section, 161 Data acquisition section, 162 Inference section

Claims

1. A meal situation estimation unit that estimates the meal situation of the person under surveillance from utensil information including a utensil image obtained by extracting from the image a utensil area that is an area of a utensil detected from an image of the table of the person under surveillance, and information regarding the position of the utensil on the table; A situation requiring attention determination unit that determines whether the situation of the person under surveillance corresponds to a situation requiring attention by using the meal situation and model data indicating a meal situation determined to be normal; A meal monitoring device comprising: The device.

2. The meal situation estimation unit estimates the meal situation by using at least one of the color, shape, size, or number of the utensils estimated from the utensil information. The meal monitoring device according to claim 1.

3. The meal situation is a situation when the person under surveillance has a meal, indicated by at least any one of the number of people having a meal, the number of items, the seats used, the person who had the meal, the utensils used, the length of the meal time, or the amount of food eaten. The meal monitoring device according to claim 1.

4. The situation requiring attention is a situation including at least any one of irregularity of the meal time, change in the length of the meal time, decrease in the number of items, decrease in the number of people having a meal, decrease in the number of times the utensils are used, or increase or decrease in the amount of food eaten. The meal monitoring device according to claim 1.

5. An image acquisition unit that acquires an image of the table of the person under surveillance; A utensil area detection unit that detects a utensil area that is an area of a utensil from the image; A utensil information creation unit that creates utensil information including a utensil image obtained by extracting the detected utensil area from the image and information regarding the position of the utensil on the table; A meal situation estimation unit that estimates the meal situation of the person under surveillance from the utensil information; A situation requiring attention determination unit that compares the meal situation with model data indicating a meal situation determined to be normal and determines whether the situation of the person under surveillance corresponds to a situation requiring attention; A notification unit that notifies a pre-registered notification destination when the situation requiring attention determination unit determines that the situation corresponds to a situation requiring attention. A meal monitoring system comprising:

6. An image acquisition step of acquiring an image of the table of the person under surveillance; A utensil area detection step of detecting a utensil area that is an area of a utensil from the image; A utensil information creation step of creating utensil information including a utensil image obtained by extracting the detected utensil area from the image and information regarding the position of the utensil on the table; A meal situation estimation step of estimating the meal situation of the person under surveillance from the utensil information; A situation - requiring - attention determination step of comparing the dietary situation with model data indicating a normal dietary situation to determine whether the situation of the person under surveillance corresponds to a situation - requiring - attention; A dietary surveillance method comprising the above.

7. In the situation - requiring - attention determination step, when it is determined that the situation of the person under surveillance corresponds to a situation - requiring - attention, further comprising a notification step of notifying a pre - registered notification destination. The dietary surveillance method according to Claim 6.

8. A dietary situation estimation step of estimating the dietary situation of the person under surveillance from utensil information including a utensil image obtained by extracting from the image a utensil area which is an area of a utensil detected from an image of the dining table of the person under surveillance and information regarding the position of the utensil on the dining table; A situation - requiring - attention determination step of comparing the dietary situation with model data indicating a normal dietary situation to determine whether the situation of the person under surveillance corresponds to a situation - requiring - attention; A dietary surveillance program for causing a computer to execute the above.

9. In the situation - requiring - attention determination step, when it is determined that the situation of the person under surveillance corresponds to a situation - requiring - attention, further comprising a notification step of notifying a pre - registered notification destination. The dietary surveillance program according to Claim 8.

10. A data acquisition unit that acquires learning data including a utensil image obtained by extracting from the image a utensil area which is an area of a utensil detected from an image of the dining table of the person under surveillance and information regarding the position of the utensil on the dining table, and the dietary situation when the image was acquired; A model generation unit that generates a learned model for inferring the dietary situation from the utensil information using the learning data; A learning device comprising the above.

11. A data acquisition unit that acquires utensil information including a utensil image obtained by extracting from the image a utensil area which is an area of a utensil detected from an image of the dining table of the person under surveillance and information regarding the position of the utensil on the dining table; An inference unit that outputs the dietary situation of the person under surveillance from the utensil information of the person under surveillance acquired by the data acquisition unit using a learned model for inferring the dietary situation from the utensil information; An inference device comprising the above.

Images