Food management method, program, food management system, and food management tag

Abstract

In this food management method, a food management tag which includes an identifier associated with food is attached to food or a food-storing storage container, and a captured image is obtained by capturing an image of said food or storage container, using an imaging element (S101). In this food management method, the food or storage container to which the food management tag is attached is recognized on the basis of the captured image that was acquired (S102, S103). In this food management method, the remaining quantity of food is estimated on the basis of the size of the recognized food management tag (S104). In this food management method, remaining quantity information is output, such information regarding the estimated remaining quantity of food (S105).

Description

Food management method, program, food management system, and food management tag

[0001] The present disclosure relates to a food management method, a program, a food management system, and a food management tag for managing food.

[0002] Patent Document 1 discloses a determination device that determines whether an item (food) is stored in or removed from a storage device (refrigerator).

[0003] JP 2016-023584 A

[0004] The present disclosure provides a food management method and the like that makes it easy to manage food.

[0005] A food management method according to one aspect of the present disclosure includes acquiring an image of a food or a storage container for storing the food, the food having a food management tag with an identifier associated with the food, captured by an imaging element. The food management method also recognizes the food or the storage container with the food management tag based on the acquired image. The food management method also estimates the remaining amount of the food based on the size of the recognized food management tag. The food management method also outputs remaining amount information related to the estimated remaining amount of the food.

[0006] A program according to one aspect of the present disclosure causes one or more processors to execute the food management method.

[0007] A food management system according to one aspect of the present disclosure includes an acquisition unit, a recognition unit, an estimation unit, and an output unit. The acquisition unit acquires an image captured by an imaging element of a food or a storage container for storing the food, the food or the storage container having a food control tag with an identifier associated with the food. The recognition unit recognizes the food or the storage container having the food control tag based on the image acquired by the acquisition unit. The estimation unit estimates the remaining amount of the food based on the size of the food control tag recognized by the recognition unit. The output unit outputs remaining amount information regarding the remaining amount of the food estimated by the estimation unit.

[0008] A food management tag according to one aspect of the present disclosure is a food management tag that is attached to food or a storage container for storing the food, has an identifier that is linked to the food, and has a variable portion that can change size depending on the amount of food remaining.

[0009] The food management method and the like according to the present disclosure has the advantage of making it easy to manage food.

[0010] FIG. 1 is a block diagram showing the overall configuration including a food management system in embodiment 1. FIG. 2 is a diagram showing a first example of use of a food management tag in embodiment 1. FIG. 3 is a diagram showing a second example of use of a food management tag in embodiment 1. FIG. 4 is a flowchart showing an example of operation of a food management system in embodiment 1. FIG. 5 is a block diagram showing the overall configuration including a food management system in a variation of embodiment 1. FIG. 6 is a block diagram showing the overall configuration including a food management system in embodiment 2. FIG. 7 is a diagram showing an example of use of a food management tag in embodiment 2. FIG. 8 is a diagram showing an example of a food management tag in embodiment 2. FIG. 9 is a flowchart showing an example of operation of a food management system in embodiment 2.

[0011] (Findings that Form the Basis of the Present Disclosure) First, the inventor's point of view will be explained below.

[0012] There is known a technology for recognizing food by capturing an image including the food and performing appropriate image recognition processing on the captured image, such as the technology disclosed in Patent Document 1. When such a technology is used to manage food stored in, for example, a refrigerator, the following problems arise.

[0013] That is, the shape and size of food change depending on how it is used, for example, when part of it is used in cooking. Therefore, the above technology has a problem in that it is difficult to determine whether unused food and food that has been partly used in cooking are the same food. Furthermore, since it is difficult to determine whether food is the same before and after use, it naturally has a problem in that it is difficult to grasp the remaining amount of food.

[0014] One possible solution is to identify food using RFID (Radio Frequency Identifier) ​​technology by attaching an RF (Radio Frequency) tag to the food. However, RF tags have not yet become widespread, making it impractical to attach RF tags to all food. Another problem is that even if RF tags are attached to food, it is still not possible to determine the amount of food remaining.

[0015] In view of the above, the inventors have come up with the present disclosure.

[0016] Hereinafter, each embodiment will be described in detail with reference to the accompanying drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0017] The inventor provides the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and does not intend for them to limit the subject matter described in the claims.

[0018] (Embodiment 1) [1-1. Overall Configuration] First, the overall configuration including a food management system 100 in embodiment 1 will be described using FIG. 1. FIG. 1 is a block diagram showing the overall configuration including a food management system 100 in embodiment 1. In embodiment 1, food management system 100 is used to manage the remaining amount of food 3 stored in refrigerator 2, in other words, the amount of food 3 used. For example, food 3 may be stored directly in refrigerator 2, or may be stored in refrigerator 2 in a state where it is stored in a storage container 5 (see FIG. 3) that preserves food 3. Furthermore, in embodiment 1, food 3 or storage container 5 is stored in refrigerator 2 with a food management tag 4 attached.

[0019] The food control tag 4 is attached to the food 3 or a storage container 5 that stores the food 3, and has an identifier that is linked to the food 3. The identifier here may be a specific character string such as a so-called ID (identifier) ​​number, or may simply be a unique feature that can be distinguished from other food control tags. More specifically, the identifier may be in a form that allows the results of an image recognition process performed on an image of the food control tag in the recognition unit 112 of the food control system 100 (described later) to be distinguished from the results of an image recognition process performed on images of other food control tags.

[0020] In the first embodiment, the food control tag 4 is, for example, a sticky note shaped like a cat, and has written thereon a character string (see FIG. 2; in this case, the character string "cabbage") indicating the name of the food 3. In other words, in the first embodiment, the identifier of the food control tag 4 is a combination of the shape of the food control tag 4 and the character string written on the food control tag 4.

[0021] The identifier of the food management tag 4 may be, for example, the character string written on the food management tag 4, the shape of the food management tag 4, or the pattern applied to the food management tag 4. The identifier of the food management tag 4 may also be a combination of two or more of the shape, pattern, and character string. In the first embodiment, the food management tag 4 may be an RF tag.

[0022] The food control tag 4 is attached to the food 3 or the storage container 5, for example, by sticking the surface coated with adhesive to any location on the food 3 or the storage container 5. The food control tag 4 may be stuck directly to the food 3 or the storage container 5, for example, as described above, or may be attached to the food 3 or the storage container 5 using, for example, a rubber band or a clip. The food control tag 4 may also be stuck to the packaging of the food 3, for example.

[0023] The refrigerator 2 includes an image sensor 21, a communication interface (hereinafter referred to as a communication I / F (Interface)) 22, a processor 23, and a storage 24. The refrigerator 2 may also include a refrigerator compartment with a door, or a drawer that forms a vegetable compartment. In the first embodiment, the image sensor 21, the communication I / F 22, the processor 23, and the storage 24 are provided in the refrigerator 2, but they may also be provided in the periphery of the refrigerator 2 as devices separate from the refrigerator 2.

[0024] The imaging element 21 is an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and is used to capture an image of the food 3 or storage container 5 to which the food control tag 4 is attached. The imaging element 21 is installed, for example, inside the refrigerator 2, on the door of the refrigerator 2, or on a member protruding from the top plate of the refrigerator 2 toward the door.

[0025] The imaging element 21 may be provided inside or outside the refrigerator 2. The imaging element 21 does not have to be pre-installed in the refrigerator 2 and may be added later. The imaging element 21 may be portable by the user. In this case, the user may hold the imaging element 21 in their hand to capture an image of the food management tag 4. The imaging element 21 may be independent or may be incorporated into some device, such as a mobile terminal with an imaging function.

[0026] Furthermore, there may be a plurality of image capturing elements 21, such as an image capturing element 21 for a refrigerator compartment, an image capturing element 21 for a freezer compartment, or an image capturing element 21 for a drawer. When there are a plurality of image capturing elements 21, the processor 23 may determine which image capturing element 21 is to be used to capture an image.

[0027] Furthermore, the imaging element 21 may be used not only to capture images of the food 3 or storage container 5 to which the food management tag 4 is attached as described below, but also to capture images of other objects, such as a user. When the imaging element 21 is used to capture an image of a user, it is possible to identify the user based on the image of the user.

[0028] The imaging element 21 captures an image of the food 3 or storage container 5 with the food control tag 4 attached, for example, when the user opens or closes the door of the refrigerator 2. Specifically, the imaging element 21 may capture an image of the food 3 or storage container 5 with the food control tag 4 attached while the door of the refrigerator 2 is open. The imaging element 21 may also capture an image of the food 3 or storage container 5 with the food control tag 4 attached that is outside the refrigerator 2 for a certain period of time after detecting that the door of the refrigerator 2 is closed. The imaging element 21 may also capture an image of the food 3 or storage container 5 with the food control tag 4 attached that is inside the refrigerator 2, for example, a certain period of time after detecting that the door of the refrigerator 2 is closed. Alternatively, the imaging element 21 may capture images at all times, regardless of whether the refrigerator 2 is opened or closed. Therefore, imaging by the imaging element 21 is performed when the user places the food 3 or storage container 5 into the refrigerator 2 (storage space) or when the user removes the food 3 or storage container 5 from the refrigerator 2.

[0029] The communication I / F 22 communicates with the communication I / F 10 of the food management system 100 via an external network NT1 such as the Internet. The communication between the communication I / F 22 and the communication I / F 10 of the food management system 100 may be wired or wireless. There are no particular limitations on the standard of communication between the communication I / F 22 and the food management system 100. When the imaging element 21 captures an image of the food 3 or storage container 5 with the food management tag 4 attached, the communication I / F 22 transmits data of the captured image to the communication I / F 10 of the food management system 100.

[0030] The processor 23 controls the operation of the refrigerator 2. The processor 23 includes a memory, and is configured to perform various functions by executing programs stored in the memory.

[0031] The storage 24 is a storage device that stores information (such as computer programs) required for the processor 11 to execute various functions. The storage 12 is realized by, for example, a semiconductor memory, but is not particularly limited and any known electronic information storage means can be used. The storage 12 stores, for example, data of images captured by the image sensor 21.

[0032] [1-2. Food Management System] Next, details of food management system 100 will be described. In the first embodiment, food management system 100 is configured with server 1 that is installed in a remote location away from the facility where refrigerator 2 is installed (here, the residence where the user lives). Note that food management system 100 may further include refrigerator 2 in addition to server 1.

[0033] In the first embodiment, the food management system 100 is described as targeting one refrigerator 2, but the food management system 100 may target multiple refrigerators 2.

[0034] As shown in FIG. 1 , the food management system 100 (server 1) includes a communication I / F 10, a processor 11, and a storage 12.

[0035] As already mentioned, the communication I / F 10 has a function to communicate with the communication I / F 22 of the refrigerator 2 via the external network NT1. The communication I / F 10 receives data of an image captured by the image sensor 21 and transmitted from the communication I / F 22 of the refrigerator 2. The captured image data received by the communication I / F 10 is provided to the processor 11.

[0036] Processor 11 is equipped with a memory and is configured to perform various functions by executing programs stored in the memory. Specifically, processor 11 functions as an acquisition unit 111, a recognition unit 112, an estimation unit 113, and an output unit 114 by executing the programs stored in the memory. In other words, food control system 100 is equipped with acquisition unit 111, recognition unit 112, estimation unit 113, and output unit 114.

[0037] The acquisition unit 111 acquires an image of the food 3 or storage container 5 to which the food management tag 4 is attached, captured by the image sensor 21. In the first embodiment, the acquisition unit 111 acquires the captured image data transmitted from the communication I / F 22 of the refrigerator 2 via the communication I / F 10.

[0038] The recognition unit 112 recognizes the food 3 or storage container 5 to which the food control tag 4 is attached, based on the captured image acquired by the acquisition unit 111. Specifically, the recognition unit 112 recognizes the image of the portion of the food control tag 4 included in the captured image and the image of the portion of the food 3 or storage container 5 included in the captured image by performing appropriate image recognition processing on the captured image acquired by the acquisition unit 111. Here, the recognition unit 112 recognizes the portion of the food control tag 4 included in the captured image as the identifier of the food control tag 4. The identifier of the food control tag 4 recognized by the recognition unit 112 and the image of the portion of the food 3 or storage container 5 recognized by the recognition unit 112 are stored in the storage 12 as data linked to each other. As a result, the identifier of the food control tag 4 and the image of the portion of the food 3 or storage container 5 to which the food control tag 4 is attached are linked and stored in the storage 12.

[0039] In the first embodiment, the recognition unit 112 further recognizes the type of food 3 based on the captured image acquired by the acquisition unit 111. Therefore, in the first embodiment, the identifier of the food management tag 4 is linked to an image of the food 3 or the portion of the storage container 5 to which the food management tag 4 is attached, and is further linked to the type of food 3 and stored in the storage 12. The recognition unit 112 can recognize the type of food 3, for example, by using a trained model that has been machine-learned to input an image of the food 3 and output the type of food 3.

[0040] The estimation unit 113 estimates the remaining amount of food 3 based on the size of the food control tag 4 recognized by the recognition unit 112. In the first embodiment, the estimation unit 113 estimates the remaining amount of food 3 by comparing the size of the food control tag 4 recognized by the recognition unit 112 with the size of the food 3 or the size of the storage container 5 recognized by the recognition unit 112. Specifically, the estimation unit 113 estimates the remaining amount of food 3 based on the ratio of the area of ​​the area recognized as the food 3 or the storage container 5 in the captured image to the area of ​​the area recognized as the food control tag 4 in the captured image. Note that the area of ​​the area here may be the number of pixels included in the area. The remaining amount of food 3 estimated by the estimation unit 113 is stored in the storage 12 in association with the identifier of the food control tag 4 and the food 3 to which the food control tag 4 is attached.

[0041] The remaining amount of food 3 may be stored in storage 12 in association with the registration date of the food 3 or the update date on which the remaining amount of the food 3 was updated. The remaining amount of food 3 may also be stored in storage 12 in association with the storage location in the refrigerator 2. Furthermore, if there are multiple image capture elements 21, the remaining amount of food 3 may be stored in storage 12 in association with information specifying which image capture element 21 was used to capture the image.

[0042] Furthermore, if the user can be identified based on the image captured by the imaging element 21, the remaining amount of food 3 may be linked to the user's identifier and stored in the storage 12. In this case, even if the user is away from the refrigerator 2, for example, when the user is out, the user can view the above-mentioned various information, including the remaining amount of food 3 linked to the user's identifier, using an information terminal such as a smartphone carried by the user. Furthermore, when viewing the above-mentioned various information, the user can modify at least a part of the above-mentioned various information by operating the information terminal.

[0043] Furthermore, for example, when only the user's identifier is linked to the remaining amount of food 3 and stored in storage 12, additional information such as the above-mentioned registration date, update date, storage location in refrigerator 2, or information specifying which imaging element 21 was used to capture the image may be further linked.

[0044] That is, in the first embodiment, the size of the food management tag 4 remains constant, and therefore, naturally, the size remains constant before and after the food 3 is used. On the other hand, the size of the food 3 decreases each time it is used, for example, in cooking. Similarly, when the food 3 is used in cooking, for example, the size of the storage container 5 may also decrease as the food 3 is transferred to a smaller storage container. Therefore, it is possible to estimate the remaining amount of food 3 by calculating the ratio of the size of the food 3 or the size of the storage container 5 to the size of the food management tag 4.

[0045] Fig. 2 is a diagram showing a first example of use of the food management tag 4 in embodiment 1. Fig. 2 shows an image of a food 3 to which a food management tag 4 is attached, captured by an imaging element 21. In the example shown in Fig. 2, the food 3 is cabbage, and the food management tag 4 is a sticky note shaped like a cat with the word "cabbage" written on it. In the image shown in Fig. 2(a), the food 3 is unused. In the image shown in Fig. 2(b), half of the food 3 has been used in cooking, etc.

[0046] For example, assume that the acquisition unit 111 acquires the captured image shown in Fig. 2(a) when the user first places food 3 in the refrigerator 2. In this case, the estimation unit 113 estimates that the remaining amount of food 3 is 100%. The estimation unit 113 also calculates "α" as the ratio of the area recognized as food 3 to the area recognized as food control tag 4 in the captured image shown in Fig. 2(a). This ratio is linked to the identifier of the food control tag 4 and the food 3 to which the food control tag 4 is attached, and stored in the storage 12.

[0047] For example, suppose that the user removes food 3 with the same food management tag 4 from the refrigerator 2 and then puts the food 3 back into the refrigerator 2, at which point the acquisition unit 111 acquires the captured image shown in Figure 2(b). In this case, the estimation unit 113 calculates the ratio of the area recognized as food 3 to the area recognized as food management tag 4 in the captured image shown in Figure 2(b) as "α / 2." The estimation unit 113 then estimates that the remaining amount of food 3 is 50% by comparing this ratio with "α," which is the ratio when the remaining amount of food 3 read from the storage 12 is 100%.

[0048] Here, the distance of the food 3 with the food control tag 4 attached to it from the imaging element 21 may change each time an image is captured by the imaging element 21. Therefore, the area of ​​the area recognized as the food control tag 4 in the captured image and the area of ​​the area recognized as the food 3 may change each time an image is captured by the imaging element 21. In the first embodiment, the estimation unit 113 utilizes the fact that the size of the food control tag 4 is constant to estimate the remaining amount of food 3 based on the ratio of the area recognized as the food 3 to the area recognized as the food control tag 4 in the captured image. Therefore, it is possible to estimate the remaining amount of food 3 even if the area of ​​the area recognized as the food control tag 4 and the area recognized as the food 3 in the captured image change each time an image is captured by the imaging element 21.

[0049] Fig. 3 is a diagram showing a second example of use of the food management tag 4 in embodiment 1. Fig. 3 shows an image captured by the imaging element 21 of a storage container 5 to which a food management tag 4 is attached. In the example shown in Fig. 3, the food 3 is green beans and is contained in the storage container 5. In the example shown in Fig. 3, the food management tag 4 is a sticky note shaped like a cat with the word "green beans" written on it. In the image shown in Fig. 3(a), the food 3 is unused and is contained in a first storage container 51. In the image shown in Fig. 3(b), half of the food 3 has been used in cooking or the like and is contained in a second storage container 52 that is half the size of the first storage container.

[0050] For example, assume that the acquisition unit 111 acquires the captured image shown in FIG. 3A when the user places a storage container 5 (first storage container 51) in the refrigerator 2 for the first time. In this case, the estimation unit 113 estimates that the remaining amount of food 3 is 0%. The estimation unit 113 also calculates "β" as the ratio of the area recognized as the storage container 5 (first storage container 51) to the area recognized as the food control tag 4 in the captured image shown in FIG. 3A. This ratio is linked to the identifier of the food control tag 4 and the food 3 to which the food control tag 4 is attached, and stored in the storage 12.

[0051] For example, suppose that the user removes a storage container 5 (first storage container 51) with the same food management tag 4 as above from the refrigerator 2, then transfers the food 3 to another storage container 5 (second storage container 52), and places the storage container 5 back into the refrigerator 2. At this time, the acquisition unit 111 acquires the captured image shown in FIG. 2B. In this case, the estimation unit 113 calculates the ratio of the area recognized as the storage container 5 (second storage container 52) to the area recognized as the food management tag 4 in the captured image shown in FIG. 3B as "β / 2." The estimation unit 113 then estimates that the remaining amount of food 3 is 50% by comparing this ratio with the ratio "β" that is used when the remaining amount of food 3 read from the storage 12 is 0%.

[0052] Here, the distance of the storage container 5 to which the food control tag 4 is attached from the imaging element 21 may change each time an image is captured by the imaging element 21. Therefore, the area of ​​the region recognized as the food control tag 4 in the captured image and the area of ​​the region recognized as the storage container 5 may change each time an image is captured by the imaging element 21. In the first embodiment, the estimation unit 113 utilizes the fact that the size of the food control tag 4 is constant to estimate the remaining amount of food 3 based on the ratio of the area recognized as the storage container 5 to the area recognized as the food control tag 4 in the captured image. Therefore, even if the area of ​​the region recognized as the food control tag 4 and the area recognized as the storage container 5 in the captured image change each time an image is captured by the imaging element 21, it is possible to estimate the remaining amount of food 3.

[0053] It is possible that the storage container 5 is a transparent container, and the food 3 contained in the storage container 5 can be imaged by the imaging element 21. In this case, the estimation unit 113 can estimate the remaining amount of food 3 by comparing the size of the food management tag 4 recognized by the recognition unit 112 with the size of the food 3 contained in the storage container 5 recognized by the recognition unit 112.

[0054] Furthermore, there may be cases where the image sensor 21 cannot capture an image of the food 3 contained in the storage container 5 when the lid of the storage container 5 is closed. In this case, if the user captures an image of the inside of the storage container 5 with the image sensor 21 while the lid is open, the estimation unit 113 can estimate the remaining amount of food 3 by comparing the size of the food management tag 4 recognized by the recognition unit 112 with the size of the food 3 contained in the storage container 5 recognized by the recognition unit 112.

[0055] The output unit 114 outputs remaining amount information regarding the remaining amount of food 3 estimated by the estimation unit 113. The remaining amount information may include information indicating the remaining amount of food 3, with the remaining amount of food 3 being 100% when the food 3 is unused, or may include information indicating the amount of food 3 used. Furthermore, the remaining amount information may include information directly indicating the remaining amount of food 3, as well as information notifying the user, for example, whether there is a sufficient amount of food 3 remaining or only a small amount of food 3 remaining.

[0056] In the first embodiment, the output unit 114 periodically outputs (transmits) the remaining amount information to the information terminal carried by the user via the communication I / F 10, for example. Furthermore, the output unit 114 outputs (transmits) the remaining amount information to the information terminal via the communication I / F 10, for example, when the communication I / F 10 receives a request signal from the information terminal. The request signal is transmitted from the information terminal to the communication I / F 10, for example, when the user performs a predetermined operation on the information terminal. If the refrigerator 2 has a display, the output unit 114 may output (transmit) the remaining amount information to the refrigerator 2 via the communication I / F 10. In this case, the processor 23 of the refrigerator 2 may display the remaining amount information acquired via the communication I / F 22 on the display.

[0057] The storage 12 is a storage device that stores information (such as computer programs) necessary for each part of the processor 11 to execute various functions. The storage 12 is realized, for example, by a semiconductor memory, but is not particularly limited and any known electronic information storage means can be used. The storage 12 stores data linking the identifier of the food control tag 4 with the food 3 to which the food control tag 4 is attached, etc.

[0058] [2. Operation] The operation (food management method) of the food management system 100 configured as described above will be described below with reference to Figure 4. Figure 4 is a flowchart showing an example of the operation of the food management system 100 in embodiment 1. First, the acquisition unit 111 acquires a captured image of food 3 or storage container 5 to which a food management tag 4 is attached, captured by the image sensor 21 (S101). Here, the acquisition unit 111 acquires, via the communication I / F 10, data of the captured image sent from the communication I / F 22 of the refrigerator 2.

[0059] Next, the recognition unit 112 recognizes the food management tag 4 included in the captured image based on the captured image acquired by the acquisition unit 111 (S102). The recognition unit 112 also recognizes the food 3 or storage container 5 included in the captured image based on the acquired captured image (S103). The order of steps S102 and S103 is not limited to this, and they may be executed in the reverse order or simultaneously in parallel.

[0060] Next, the estimation unit 113 estimates the remaining amount of food 3 by comparing the size of the food management tag 4 recognized by the recognition unit 112 with the size of the food 3 or the size of the storage container 5 recognized by the recognition unit 112 (S104). Then, the output unit 114 outputs remaining amount information regarding the remaining amount of food 3 estimated by the estimation unit 113 (S105).

[0061] 3. Advantages, etc. Advantages of the food management system 100 (food management method) according to the first embodiment will be described below.

[0062] As already mentioned, the technology for recognizing food by capturing an image including the food and performing appropriate image recognition processing on the captured image has the problem that it is difficult to determine whether the food is the same before and after use, and it is also difficult to estimate the amount of food remaining. Also, the method of identifying food by attaching an RF tag to the food and using RFID technology has the problem that it is not possible to grasp the amount of food remaining.

[0063] In contrast, the food management system 100 (food management method) in embodiment 1 can determine whether the food 3 is the same before and after use by recognizing the food 3 or storage container 5 to which the food management tag 4 is attached. Furthermore, the food management system 100 (food management method) in embodiment 1 can estimate the remaining amount of food 3 based on the size of the recognized food management tag 4 (particularly in embodiment 1, by comparing the size of the recognized food management tag 4 with the size of the recognized food 3 or storage container 5).

[0064] Therefore, in the food management system 100 (food management method) of embodiment 1, the remaining amount of food 3 can be determined by referring to the estimated remaining amount of food 3, which has the advantage of making it easier to manage food 3 compared to simply identifying food 3.

[0065] 4. Modifications of the First Embodiment Modifications of the first embodiment will be described below.

[0066] FIG. 5 is a block diagram showing the overall configuration of a food management system 100A according to a variation of the first embodiment. The food management system 100A is composed of a server 1A. The configuration of the server 1A is the same as that of the server 1. As shown in FIG. 5, in this variation, the refrigerator 2 further includes a microphone 25. The food management system 100A according to this variation differs from the food management system 100 according to the first embodiment in that the acquisition unit 111 further acquires audio information relating to the amount of food 3 remaining, collected by the microphone 25. The food management system 100A according to this variation also differs from the food management system 100 according to the first embodiment in that the output unit 114 outputs, in the remaining amount information, information relating to the amount of food 3 remaining, based on the audio information acquired by the acquisition unit 111.

[0067] The microphone 25 picks up the voice uttered by the user, for example, when the user places the food item 3 or the storage container 5 into the refrigerator 2 (storage space) or when the user removes the food item 3 or the storage container 5 from the refrigerator 2. For example, when the user places one head of cabbage into the refrigerator 2 as the food item 3, the user utters "one head of cabbage." As a result, the acquisition unit 111 acquires the voice information "one head of cabbage" along with the captured image of the cabbage (food item 3) to which the food item control tag 4, 4A is attached. The identifier of the food item control tag 4, 4A recognized by the recognition unit 112, the recognition data of the cabbage (food item 3) recognized by the recognition unit 112, and the voice information "one head of cabbage" are stored in the storage 12 in association with each other.

[0068] The output unit 114 can then output the audio information together with the remaining amount information. That is, the output unit 114 can output remaining amount information including the recognition data of the cabbage (food 3) recognized by the recognition unit 112, the remaining amount of the cabbage (food 3) estimated by the estimation unit 113, and the audio information "one head of cabbage."

[0069] In this remaining amount information, the audio information "one cabbage" is data indicating the actual remaining amount of cabbage (food item 3). Therefore, for example, when the estimation unit 113 estimates the remaining amount of food item 3 using a trained model that has been machine-learned to input a captured image of food item 3 or storage container 5 with food management tag 4, 4A attached thereto and output the remaining amount of food item 3, this remaining amount information can be used as training data when retraining the trained model. That is, the audio information in this remaining amount information indicates the actual remaining amount of food item 3, and therefore serves as correct answer data. Therefore, the food management system 100A in this modified example has the advantage of making it easier to collect a data set for retraining in which the actual remaining amount of food item 3 is used as correct answer data.

[0070] The trained model may be machine-trained to further output the type of food 3. Even in this case, the remaining amount information can be used as training data when retraining the trained model. That is, the audio information in the remaining amount information also includes the type of food 3, and therefore becomes correct data.

[0071] (Embodiment 2) [1. Configuration] A food management system 100B in embodiment 2 will now be described with reference to Figure 6. Figure 6 is a block diagram showing the overall configuration including the food management system 100B in embodiment 2. The food management system 100B in embodiment 2 is composed of a server 1B. The configuration of server 1B is similar to that of server 1. As shown in Figure 6, the food management system 100B in embodiment 2 is premised on the fact that a food management tag 4A is attached to the food 3 or the storage container 5. The food management system 100B in embodiment 2 differs from the food management system 100 in embodiment 1 in that the estimation unit 113 estimates the remaining amount of food 3 based on the size of the variable portion 42 of the food management tag 4A recognized by the recognition unit 112.

[0072] In other words, in embodiment 2, the estimation unit 113 estimates the remaining amount of food 3 based only on the size of the food management tag 4A (here, the size of the variable portion 42) without referring to the size of the food 3 or the size of the storage container 5.

[0073] Fig. 7 is a diagram showing an example of use of the food management tag 4A in embodiment 2. Fig. 7 shows an image of a food 3 with a food management tag 4A attached, captured by an imaging element 21. In the example shown in Fig. 7, the food 3 is cabbage. In the image shown in Fig. 7(a), the food 3 is unused. In the image shown in Fig. 7(b), half of the food 3 has been used in cooking or the like.

[0074] As shown in Figure 7, the food management tag 4A in the second embodiment is a rectangular sticky note overall, and is composed of an invariable portion 41 and a variable portion 42. The invariable portion 41 has a blank space for writing, for example, a character string for the food 3, and is a portion of invariable size. The variable portion 42 has a plurality of (here, six) removable small pieces, and is a portion of variable size. The size of the variable portion 42 changes when, for example, the user removes the small pieces by tearing them off depending on the amount of food 3 remaining.

[0075] That is, the food management tag 4A has a variable portion 42 that can change its size depending on the amount of food 3 remaining. The variable portion 42 is also configured to be removable by the user depending on the amount of food 3 remaining.

[0076] In the second embodiment, the estimation unit 113 can estimate the remaining amount of food 3 based on the size of the variable portion 42 of the food management tag 4A recognized by the recognition unit 112.

[0077] Here, the processor 11 or the recognition unit 112 distinguishes and recognizes the invariant portion 41 and the variable portion 42 of the food control tag 4A in the following manner.

[0078] As an example, the invariant portion 41 is marked with a marker that can identify it as the invariant portion 41, and the multiple removable small pieces of the variable portion 42 are each marked with a marker that can identify them as small pieces. In this case, the processor 11 or the recognition unit 112 can recognize the invariant portion 41 and the multiple removable small pieces of the variable portion 42 by performing appropriate image recognition processing on the image captured by the imaging element 21.

[0079] Furthermore, the processor 11 or the recognition unit 112 only needs to be able to recognize the number of small pieces that remain unremoved from the invariant portion 41, or the area of ​​the remaining small pieces, among the multiple removable small pieces in the variable portion 42. In the former case, the processor 11 or the recognition unit 112 may recognize the number of small pieces that remain unremoved from the invariant portion 41 by counting the number of markers that can be identified as small pieces. In the latter case, the processor 11 or the recognition unit 112 may recognize the area of ​​the small pieces that remain unremoved from the invariant portion 41 from the change in the overall area of ​​the recognized small pieces. Furthermore, if each of the multiple removable small pieces in the variable portion 42 is assigned a unique feature, such as a number, the processor 11 or the recognition unit 112 may recognize which small pieces remain unremoved in the invariant portion 41.

[0080] It should be noted that there may be cases where a small piece of the variable portion 42 has been removed from the invariant portion 41 but is still within the image captured by the imaging element 21. In such cases, the processor 11 or the recognition unit 112 can recognize that the small piece has been removed from the invariant portion 41 if the small piece is not located near the invariant portion 41, for example, based on the distance between the small piece and the invariant portion 41.

[0081] As another example, the same marker is printed on the entire invariable portion 41 and the variable portion 42. In this case, the processor 11 or the recognition unit 112 may recognize the amount of change in the variable portion 42 based on the change in the area of ​​the marker recognized in the image captured by the imaging element 21. For example, it is assumed that information indicating that the area of ​​the variable portion 42 is 10% of the total area of ​​the markers in the invariable portion 41 and the variable portion 42 is stored in advance in the memory. In this case, when the area of ​​the recognized marker decreases by 1%, the processor 11 or the recognition unit 112 can estimate that the area of ​​the variable portion 42 has decreased by 10%, in other words, that 10% of the food 3 has been consumed.

[0082] The estimation of the remaining amount of food 3 by the estimation unit 113 will be specifically described below.

[0083] For example, suppose that the acquisition unit 111 acquires the captured image shown in Fig. 7(a) when the user first places food 3 in the refrigerator 2. In this case, the estimation unit 113 estimates that the remaining amount of food 3 is 100%. The estimation unit 113 also calculates the area "γ" of the variable portion 42 in the area recognized as the food control tag 4A in the captured image shown in Fig. 7(a). This area is linked to the identifier of the food control tag 4A and the food 3 to which the food control tag 4A is attached, and stored in the storage 12.

[0084] For example, suppose that a user removes food 3 with the same food management tag 4A from the refrigerator 2 and then puts the food 3 back into the refrigerator 2. At this time, the acquisition unit 111 acquires the captured image shown in FIG. 7B. Here, half of the variable portion 42 of the food management tag 4A has been removed by the user's hand, depending on the amount of food 3 remaining. The estimation unit 113 calculates the area "γ / 2" of the variable portion 42 in the area recognized as the food management tag 4A in the captured image shown in FIG. 7B. The estimation unit 113 then estimates that the amount of food 3 remaining is 50% by comparing this area with the area "γ" that would be obtained if the amount of food 3 remaining was 100% as read from the storage 12.

[0085] Furthermore, in the second embodiment, a food management tag 4B shown in FIG. 8 may be attached to the food 3 instead of the food management tag 4A. FIG. 8 is a diagram showing an example of the food management tag 4B in the second embodiment. FIG. 8 shows an image of the food 3 to which the food management tag 4B is attached, captured by the imaging element 21. In the example shown in FIG. 8, the food 3 is cabbage, and the food management tag 4B is packaging for the cabbage. Here, the packaging is manually removed or folded depending on the amount of food 3 remaining. Therefore, the entire food management tag 4B is a variable portion. Although not shown in FIG. 8, the food management tag 4B may have, as its identifier, multiple character strings indicating the name of the food 3 or the like, or may have a pattern different from other food management tags 4B. The food management tag 4B may also have, as its identifier, a one-dimensional code such as a barcode, a two-dimensional code such as a QR code (registered trademark), a color code, or the like. These codes may be written on, for example, a piece of paper with an adhesive portion, and attached to the food management tag 4B by sticking the piece of paper to the food management tag 4B. In the image shown in (a) of Figure 8, the food 3 is unused. In the image shown in (b) of Figure 8, the food 3 has been half used in cooking, etc.

[0086] For example, assume that the acquisition unit 111 acquires the captured image shown in (a) of Figure 8 when the user first places food 3 in the refrigerator 2. In this case, the estimation unit 113 estimates that the remaining amount of food 3 is 100%. The estimation unit 113 also calculates the area "δ" of the region recognized as the food control tag 4B (i.e., the variable portion) in the captured image shown in (a) of Figure 8. This area is linked to the identifier of the food control tag 4B and the food 3 to which the food control tag 4B is attached, and stored in the storage 12.

[0087] For example, suppose that the user removes food 3 with the same food management tag 4B from the refrigerator 2 and then puts the food 3 back into the refrigerator 2. At this time, the acquisition unit 111 acquires the captured image shown in FIG. 8(b). Here, the food management tag 4B is either half removed or folded by the user depending on the amount of food 3 remaining. The estimation unit 113 calculates the area "δ / 2" of the region recognized as the food management tag 4B (i.e., the variable portion) in the captured image shown in FIG. 8(b). The estimation unit 113 then estimates that the amount of food 3 remaining is 50% by comparing this area with the area "δ" read from the storage 12 when the amount of food 3 remaining is 100%.

[0088] [2. Operation] The operation (food management method) of the food management system 100B configured as described above will be described below with reference to Figure 9. Figure 9 is a flowchart showing an example of the operation of the food management system 100B in embodiment 2. First, the acquisition unit 111 acquires a captured image of food 3 or storage container 5 bearing a food management tag 4A, 4B, captured by the image sensor 21 (S201). Here, the acquisition unit 111 acquires, via the communication I / F 10, data of the captured image transmitted from the communication I / F 22 of the refrigerator 2.

[0089] Next, the recognition unit 112 recognizes the food management tags 4A, 4B included in the captured image based on the captured image acquired by the acquisition unit 111 (S202). The recognition unit 112 also recognizes the food 3 or storage container 5 included in the captured image based on the acquired captured image (S203). The order of steps S202 and S203 is not limited to this, and they may be executed in the reverse order or simultaneously in parallel.

[0090] Next, the estimation unit 113 estimates the remaining amount of food 3 based on the size of the variable portion 42 of the food management tag 4A, 4B recognized by the recognition unit 112 (S204). Then, the output unit 114 outputs remaining amount information regarding the remaining amount of food 3 estimated by the estimation unit 113 (S205).

[0091] 3. Advantages, etc. The advantages of the food management system 100B (food management method) in embodiment 2 are described below. As already mentioned, the food management system 100B (food management method) in embodiment 2 estimates the remaining amount of food 3 based solely on the size of the food management tag 4A (here, the size of the variable portion 42), without referring to the size of the food 3 or the size of the storage container 5. Therefore, the food management system 100B (food management method) in embodiment 2 has the advantage of being able to estimate the remaining amount of food 3 without knowing the size of the food 3 or the size of the storage container 5, as long as it can recognize the size of the variable portion 42 of the food management tags 4A, 4B.

[0092] 4. Modifications of the Second Embodiment Modifications of the second embodiment will be exemplified below.

[0093] In the second embodiment, the food control tag 4B is packaging that encases the food 3, but is not limited to this. For example, the food control tag may be made of edible ink. In this case, the food control tag is printed directly on the food 3. In this case, the size of the food control tag changes depending on the amount of food 3 remaining, just as in the case of packaging, so the estimation unit 113 can estimate the amount of food 3 remaining by referring to the size of the food control tag (variable portion).

[0094] Furthermore, the food control tag may be made of transparent ink. In this case, since the food control tag itself has no design, it can be printed on the entire food 3 without worrying about changing the design due to printing the food control tag on the food 3. The transparent ink may be made of a material that reacts to light having a specific wavelength other than visible light, such as infrared light. In other words, if the food control tag is made of transparent ink, it hardly reflects visible light even when it is incident, making it difficult for the human eye to recognize the food control tag. On the other hand, when light having a specific wavelength, such as infrared light, is incident on the food control tag, the light is reflected, and the food control tag can be imaged by receiving the reflected light with the imaging element 21. In this case, the imaging element 21 is configured without a filter that blocks light having a specific wavelength.

[0095] In the second embodiment, the estimation unit 113 may perform the estimation process by the estimation unit 113 in the first embodiment, in addition to the above estimation process, that is, a process of estimating the remaining amount of food 3 by comparing the size of the food management tag 4A (here, the invariant portion 41) with the size of the food 3 or the size of the storage container 5. In this case, the estimation unit 113 may use the average value of the results of both estimation processes as the remaining amount of food 3.

[0096] (Modifications) As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to the embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0097] Therefore, below, examples of modifications common to each embodiment will be given.

[0098] In each of the above embodiments, the food control tags 4, 4A, 4B may be made of a material that reflects light of a specific wavelength. Here, light of a specific wavelength is, for example, illumination light irradiated onto the interior of the refrigerator 2 or near-infrared light. In this case, when capturing an image using the image sensor 21 installed inside the refrigerator 2, the food control tags 4, 4A, 4B reflect light of a specific wavelength, making it easier to capture an image using the image sensor 21. In other words, this has the advantage that the food control tags 4, 4A, 4B can be more clearly captured using the image sensor 21 when irradiated with light of a specific wavelength.

[0099] In each of the above embodiments, the food 3 or storage container 5 is stored in the refrigerator 2, but this is not limiting. For example, the food 3 or storage container 5 may be stored in a storage space other than the refrigerator 2, such as a food storage room such as a pantry. In this case, the image sensor 21 and the communication I / F 22 may be provided in the storage space.

[0100] In each of the above embodiments, the image sensor 21 captures images when the food 3 or storage container 5 is put into or taken out of the refrigerator 2, but this is not limited to this. For example, the image sensor 21 may periodically capture images of the interior of the refrigerator 2, regardless of whether the door of the refrigerator 2 is opened or closed.

[0101] In the above-described embodiments, images captured by refrigerator 2 are transmitted to food management systems 100, 100A, and 100B, but this is not limited to this. For example, images captured by a user's information terminal, such as a smartphone, may be transmitted to food management systems 100, 100A, and 100B. In this case, the information terminal only needs to be equipped with image sensor 21 and communication I / F 22. In this case, a series of processes, such as estimating the remaining amount of food 3, performed by food management systems 100, 100A, and 100B is executed when, for example, a user operates the information terminal to transmit the captured images to food management systems 100, 100A, and 100B.

[0102] The food control tags 4, 4A, and 4B in the above embodiments can be distributed independently on the market. That is, the food control tags 4, 4A, and 4B are not accessories to the food 3 or storage container 5, but can be attached to the food 3 or storage container 5 at the user's discretion.

[0103] In addition, for example, in each of the above embodiments, food management systems 100, 100A, and 100B are implemented by servers 1, 1A, and 1B, respectively, but this is not limited to this. For example, food management systems 100, 110A, and 100B may all be implemented by an information terminal such as a smartphone carried by a user. In this case, communication I / F 10, processor 11, and storage 12 of food management systems 100, 100A, and 100B are implemented by the communication I / F, processor, and storage provided in the information terminal, respectively.

[0104] Furthermore, for example, in the above embodiments, food management systems 100, 100A, and 100B are implemented as a single device, but they may also be implemented as multiple devices. When food management systems 100, 100A, and 100B are implemented as multiple devices, the components of food management systems 100, 100A, and 100B may be distributed among the multiple devices in any manner. In other words, the present disclosure may be implemented using cloud computing or edge computing. As an example, food management systems 100, 100A, and 100B may be provided in a storage space for storing food 3 or storage containers 5, such as a refrigerator 2.

[0105] Furthermore, for example, in each of the above embodiments, all or some of the components of food control systems 100, 100A, and 100B of the present disclosure may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or processor reading and executing a software program recorded on a recording medium such as an HDD (Hard Disk Drive) or semiconductor memory.

[0106] Furthermore, the components of food control systems 100, 100A, and 100B of the present disclosure may be configured with one or more electronic circuits, each of which may be a general-purpose circuit or a dedicated circuit.

[0107] The one or more electronic circuits may include, for example, a semiconductor device, an integrated circuit (IC), or a large-scale integration (LSI). The IC or LSI may be integrated on a single chip or on multiple chips. Although the IC or LSI is referred to here as an IC or LSI, the name may vary depending on the degree of integration, and may be called a system LSI, a very large-scale integration (VLSI), or an ultra-large-scale integration (ULSI). A field programmable gate array (FPGA), which is programmed after the LSI is manufactured, can also be used for the same purpose.

[0108] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, device, method, integrated circuit, or computer program. Alternatively, the present disclosure may be realized as a computer-readable non-transitory recording medium, such as an optical disc, HDD, or semiconductor memory, on which the computer program is stored. For example, the present disclosure may be realized as a program for causing a computer to execute the proposed method in each of the above embodiments. Furthermore, the program may be recorded on a computer-readable non-transitory recording medium, such as a CD-ROM, or may be distributed via a communication channel, such as the Internet.

[0109] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed descriptions have been provided for that purpose.

[0110] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0111] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0112] (Summary) As described above, the food management method according to the embodiment acquires an image of a food 3 or a storage container 5 storing the food 3, which is attached with a food management tag 4, 4A, or 4B having an identifier associated with the food 3, captured by the image sensor 21 (S101, S201). The food management method then recognizes the food 3 or storage container 5 to which the food management tag 4, 4A, or 4B is attached based on the acquired image (S102, S103, S202, S203). The food management method then estimates the remaining amount of the food 3 based on the size of the recognized food management tag 4, 4A, or 4B (S104, S204). The food management method then outputs remaining amount information related to the estimated remaining amount of the food 3 (S105, S205).

[0113] This has the advantage that the remaining amount of food 3 can be determined by referring to the estimated remaining amount of food 3, making it easier to manage food 3 compared to simply identifying food 3.

[0114] Furthermore, for example, in the food management method, the remaining amount of food 3 is estimated by comparing the size of the recognized food management tag 4, 4A with the recognized size of the food 3 or the size of the storage container 5.

[0115] This has the advantage that the accuracy of estimating the remaining amount of food 3 is more likely to be improved compared to when only the size of food 3 is taken into consideration.

[0116] Furthermore, for example, in the food management method, the type of food 3 is further recognized based on the acquired captured image.

[0117] This has the advantage that not only the remaining amount of food 3 but also the type of food 3 can be managed, making it easier to manage food 3 in more detail.

[0118] Furthermore, for example, the food management tags 4A, 4B have a variable portion 42 that can change size depending on the remaining amount of food 3. Furthermore, in the food management method, the remaining amount of food 3 is estimated based on the size of the variable portion 42 of the recognized food management tag 4A, 4B.

[0119] This has the advantage that the remaining amount of food 3 can be estimated without recognizing the size of the food 3 or the size of the storage container 5, as long as the size of the variable portion 42 of the food management tag 4A, 4B can be recognized, thereby reducing the processing load.

[0120] Furthermore, for example, the food management method further acquires audio information relating to the remaining amount of food 3 collected by microphone 25. Furthermore, the food management method outputs information relating to the remaining amount of food 3 based on the acquired audio information, which is included in the remaining amount information.

[0121] This has the advantage that, for example, when relearning an estimation model that estimates the remaining amount of food 3 using machine learning, it becomes easier to collect a dataset for relearning that uses the actual remaining amount of food 3 as correct data.

[0122] Furthermore, for example, the imaging element 21 captures an image when the user places the food 3 or storage container 5 into the storage space (refrigerator 2) or when the user removes the food 3 or storage container 5 from the storage space.

[0123] This has the advantage that the remaining amount of food 3 can be estimated at a time when there is a high possibility that food 3 will be used, making it easier to accurately manage the remaining amount of food 3.

[0124] Furthermore, for example, the food control tags 4, 4A, and 4B are made of a material that reflects light of a specific wavelength.

[0125] This has the advantage that the food control tags 4, 4A, and 4B can be more easily and clearly imaged by the imaging element 21 when irradiated with light of a specific wavelength.

[0126] Furthermore, the program in the embodiment causes one or more processors to execute the food management method described above.

[0127] This has the advantage that the remaining amount of food 3 can be determined by referring to the estimated remaining amount of food 3, making it easier to manage food 3 compared to simply identifying food 3.

[0128] Furthermore, the food management systems 100, 100A, and 100B according to the embodiments include an acquisition unit 111, a recognition unit 112, an estimation unit 113, and an output unit 114. The acquisition unit 111 acquires captured images, captured by the image sensor 21, of food 3 or a storage container 5 for storing food 3, each of which has a food management tag 4, 4A, or 4B attached thereto and which has an identifier associated with the food 3. The recognition unit 112 recognizes the food 3 or the storage container 5 to which the food management tag 4, 4A, or 4B is attached based on the captured images acquired by the acquisition unit 111. The estimation unit 113 estimates the remaining amount of food 3 based on the size of the food management tag 4, 4A, or 4B recognized by the recognition unit 112. The output unit 114 outputs remaining amount information regarding the remaining amount of food 3 estimated by the estimation unit 113.

[0129] This has the advantage that the remaining amount of food 3 can be determined by referring to the estimated remaining amount of food 3, making it easier to manage food 3 compared to simply identifying food 3.

[0130] In addition, the food management tags 4A and 4B in the embodiment are food management tags that are attached to food 3 or a storage container 5 for storing food 3, have an identifier linked to food 3, and have a variable portion 42 that can change size depending on the remaining amount of food 3.

[0131] This has the advantage that by attaching the food management tags 4A, 4B to the food 3 or the storage container 5, it becomes possible to grasp the remaining amount of food 3, making it easier to manage the food 3 compared to when a tag for simply identifying the food 3 is attached to the food 3 or the storage container 5.

[0132] Furthermore, for example, the variable portion 42 is configured to be removable by the user depending on the amount of food 3 remaining.

[0133] This has the advantage that the user can change the size of the variable portion 42 according to the remaining amount of food 3, which makes it easier to improve the accuracy of estimating the remaining amount of food 3 based on the size of the variable portion 42.

[0134] The present disclosure is applicable to systems for managing food products, etc.

[0135] REFERENCE SIGNS LIST 1, 1A, 1B Server 100, 100A, 100B Food management system 10 Communication I / F 11 Processor 111 Acquisition unit 112 Recognition unit 113 Estimation unit 114 Output unit 12 Storage 2 Refrigerator (storage space) 21 Image sensor 22 Communication I / F 23 Processor 24 Storage 25 Microphone 3 Food 4, 4A, 4B Food management tag 41 Invariant part 42 Variable part 5 Storage container 51 First storage container 52 Second storage container NT1 External network

Claims

1. Obtaining an image of a food or a storage container for storing the food, the food being tagged with a food management tag having an identifier associated with the food, by capturing the image of the food with an imaging element; Based on the acquired captured image, the food management tag and the food or the storage container to which the food management tag is attached are recognized; estimating the remaining amount of the food based on the area of ​​the recognized food management tag and the area of ​​the recognized food or storage container; outputting remaining amount information regarding the estimated remaining amount of the food; Food management methods.

2. A food management tag having an identifier linked to a food, the food having a variable portion capable of changing its size according to the amount of the food remaining, an image of the food or a storage container for storing the food, the image being captured by an image sensor; Recognizing the food or the storage container to which the food management tag is attached based on the acquired captured image; estimating the remaining amount of the food based on the size of the variable portion of the recognized food management tag; outputting remaining amount information regarding the estimated remaining amount of the food; The movable portion is configured to be removable by a user depending on the remaining amount of the food. Food management methods.

3. Further recognizing the type of the food based on the acquired captured image. The food management method according to claim 1 or 2.

4. the food management tag has a variable portion capable of changing a size according to a remaining amount of the food, estimating the remaining amount of the food based on the size of the variable portion of the recognized food management tag; The food management method according to claim 1 or 2.

5. Further acquiring audio information regarding the remaining amount of the food collected by a microphone; outputting information about the remaining amount of the food based on the acquired voice information together with the remaining amount information; The food management method according to claim 1 or 2.

6. The imaging by the imaging element is performed at a timing when a user puts the food or the storage container into the storage space or when the user removes the food or the storage container from the storage space. The food management method according to claim 1 or 2.

7. The food control tag is formed of a material that reflects light of a specific wavelength. The food management method according to claim 1 or 2.

8. The food control tag is made of a transparent ink. The food management method according to claim 1 or 2.

9. The remaining amount information is further linked to a storage location of the food. The food management method according to claim 1 or 2.

10. One or more processors, Executing the food management method according to claim 1 or 2, program.

11. an acquisition unit that acquires an image of a food product or a storage container for storing the food product, the image being captured by an image sensor, the food product being tagged with a food product management tag having an identifier associated with the food product; a recognition unit that recognizes the food management tag and the food or the storage container to which the food management tag is attached based on the captured image acquired by the acquisition unit; an estimation unit that estimates the remaining amount of the food based on an area of ​​the food management tag recognized by the recognition unit and an area of ​​the food or the storage container recognized by the recognition unit; and an output unit that outputs remaining amount information regarding the remaining amount of the food estimated by the estimation unit. Food control system.

12. A food management tag having an identifier linked to a food, the food management tag having a variable portion capable of changing its size according to the amount of food remaining, an acquisition unit that acquires an image of the food or a storage container for storing the food, the image being captured by an image sensor; a recognition unit that recognizes the food or the storage container to which the food management tag is attached based on the captured image acquired by the acquisition unit; an estimation unit that estimates the remaining amount of the food based on the size of the variable portion of the food management tag recognized by the recognition unit; and an output unit that outputs remaining amount information regarding the remaining amount of the food estimated by the estimation unit. Food control system.

13. A food management tag attached to a food or a storage container for storing the food and having an identifier associated with the food, A variable portion capable of changing its size according to the remaining amount of the food. Food control tag.

14. The movable portion is configured to be removable by a user depending on the remaining amount of the food. The food control tag of claim 13.