Detecting amount remaining of food items in a household appliance

Abstract

The present disclosure relates to a method for detecting an amount remaining of food items (14) located in a household appliance (10), and a device (16) performing the method. The method comprising acquiring (S101) a visual representation (19a) of at least one food item (14) located in the household appliance (10), determining (S102), from the acquired visual representation (19a), a category of preconfigured food items (21a-21d) to which said at least one food item (14) belongs, and determining (S103) an amount of said at least one food item (14) remaining with respect to a complete and non-consumed representation of the food item (14).

Description

DETECTING AMOUNT REMAINING OF FOOD ITEMS IN A HOUSEHOLD APPLIANCETECHNICAL FIELD

[0001] The present disclosure relates to a method for detecting an amount remaining of food items located in a household appliance, and a device performing the method.BACKGROUND

[0002] In a household appliance such as a refrigerator or a wine fridge, a user will take out food items such as bottles, fish, meat, vegetables, fruit, etc., consume a part of the food item and thereafter put it back in the fridge.

[0003] In order for the user to know which food items have been partly consumed, such as a pizza having been partly eaten or a bottle of wine having been partly drunk, the user will have to open the fridge and visually inspect the food items.

[0004] In the art, there are solutions where partly consumed food items are weighed in the fridge to determine that the food items indeed have been partly consumed, but such solutions require sensors in the form of scales being installed in the fridge. There is thus room for improvement.SUMMARY

[0005] One objective is to solve, or at least mitigate, this problem in the art and thus to provide an improved method of detecting amount remaining of partly consumed food items located in a household appliance-

[0006] In a first aspect, this is attained by a method for detecting an amount remaining of food items located in a household appliance. The method comprises acquiring a visual representation of at least one food item located in the household appliance, determining, from the acquired visual representation, a category of preconfigured food items to which said at least one food item belongs, and determining an amount of said at least one food item remaining with respect to a complete and non-consumed representation of the food item.

[0007] In a second aspect, this is attained by a device configured to detect amount remaining of food items located in a household appliance. The device is operative toacquire a visual representation of at least one food item located in the household appliance, determine, from the acquired visual representation, a category of preconfigured food items to which said at least one food item belongs, and determine an amount of said at least one food item remaining with respect to a complete and non-consumed representation of the food item.

[0008] Advantageously, the user is made aware of the amount remaining of selected food items being stored in a household appliance such as a fridge. By comparing a food item being represented in e.g. an image captured by a camera in the household appliance with a previously captured image of the same type of food item when not having been consumed, it is possible to determine an amount remaining of the food item.

[0009] In an embodiment, the determining of a category of food items to which said at least one food item belongs comprises comparing the acquired visual representation with preconfigured food items, and if there is a match, the food item of the acquired visual representation is considered to belong to a same category of food items as the preconfigured food item for which there is a match.Advantageously, a food item such as e.g. a pizza, is compared to a preconfigured representation of the same (unconsumed) food item, and if there is a match it can be concluded how much of the food item remains.

[0010] In an embodiment, the determining of an amount of said at least one food item remaining comprises, in case the food item is represented by a container accommodating fluid, determining a level of fluid remaining in the bottle and estimating, from the determined level and the determined category of containers to which the container belongs, an amount of fluid remaining in the container. Advantageously, this embodiment enables determining level of fluid (e.g. wine) remaining in a container such as a bottle.

[0011] In an embodiment, the determining of an amount of said at least one food item remaining comprises determining an area of the food item in the acquired visual representation and estimating, from the determined area and a determined area of a food item of the determined category to which the food item belongs, an amount remaining of the food item.

[0012] In an embodiment, the determining of an amount of said at least one food item remaining comprises, in case a group of food items of the same category is determined to be present in the acquired visual representation, comprises determining that the number of food items in the group is less than an originally registered number of food items for the group. Advantageously, a number of food items remaining out of a group of food items originally being placed in a fridge can be determined.

[0013] In an embodiment, a projective transformation is performed of a pose of the at least one food item in the acquired visual representation to create a new representation of the at least one food item having a pose being aligned with a pose of the preconfigured food items with which the food item is compared for finding a match. Advantageously, this may facilitate comparison of food items.

[0014] In an embodiment, image segmentation is performed on the acquired visual representation for creating a representation where the food item has a higher contrast.

[0015] In an embodiment, the performing of image segmentation comprises applying a binary mask to the acquired the acquired visual representation.

[0016] In an embodiment, the user is alerting of an estimated time instance when the food item should be consumed.

[0017] In an embodiment, the user is alerted of an exact location where the food item is located in the household appliance.

[0018] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, in which:

[0020] Figure i shows a front perspective view of a prior art household appliance in which embodiments may be implemented;

[0021] Figure 2 shows the appliance of Figure 1 where the door has being opened such that a user can reach any bottle stored therein;

[0022] Figure 3 illustrates a household appliance arranged with a camera according to an embodiment;

[0023] Figure 4 shows a flowchart illustrating a method of detecting amount remaining of partly consumed food items located in a household appliance according to an embodiment;

[0024] Figure 5 illustrates a control unit acquiring a representation of a bottle using the camera according to an embodiment;

[0025] Figure 6 illustrate different categories of bottles with which the appliance is preconfigured according to an embodiment;

[0026] Figure 7 illustrates the user being alerted of determined amount of wine remaining in a bottle via a display or a smart phone of the user according to an embodiment;

[0027] Figure 8 shows a front view of a regular fridge in which embodiments may be implemented;

[0028] Figure 9 illustrates detection of amount remaining of a pizza in a refrigerator according to an embodiment;

[0029] Figure 10 illustrates a device according to an embodiment.DETAILED DESCRIPTION

[0030] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown.

[0031] These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0032] Figure 1 shows a front perspective view of a prior art household appliance 10 configured to store food items, such as for instance fruit and vegetables, canned goods, milk cartons, meat, cheese, bottles, in which embodiments maybe implemented. The household appliance 10 will in the following example be embodied in the form of a wine cabinet, i.e. a wine refrigerator storing food items in the form of wine bottles. However, embodiments will also be described relating to food items of a regular fridge.

[0033] Now, the fridge 10 comprises a cabinet 11 in which the food items, exemplified in the form of wine bottles, are stored. The fridge 10 comprises a front door 12 for opening and closing the fridge 10. In the case of a wine fridge, a main part of the door 12 is typically made of clear or slightly tinted glass enclosed by a section of darkly tinted glass.

[0034] Figure 2 shows the fridge where the door 12 is being opened such that a user can reach any bottle stored therein.

[0035] As shown, the cabinet 11 comprises a plurality of retainers 13 in the form of shelves on which the bottles 14 are stored. It is envisaged that an interior of the cabinet 11 may comprise another couple of shelves for further bottle storage or, as shown in Figure 2, that a bottom section of the interior of the cabinet 11 is free from shelves such that larger items may be stored on a cabinet floor, e.g. magnum bottles.

[0036] The wine cabinet 11 may be arranged with different temperature zones; for instance, a top section maybe arranged to store wine at a first temperature, while a bottom section maybe arranged to store wine at a second temperature, where the two sections are thermally separated from each other. Commonly, white wines are stored at a slightly lower temperature than red wines.

[0037] Now, assuming that a user selects a bottle 14 from the fridge 10, opens a bottle 14, drinks some of the wine therein and then puts the bottle 14 back on one of the retainers 13. If the user wants to finish the bottle 14 at a later occasion, say a couple of days later, she will have to memorize where she placed the bottle 14 in the fridge 10, or tediously go over the bottles to see which one has already been opened and drunk from, since it is typically difficult to see with the naked eye which one of the bottles partly has been consumed.

[0038] Figure 3 illustrates a fridge 10 according to an embodiment wherein to resolve this issue, the fridge 10 is arranged with a camera 15, for instance arranged on an inside of the door 12, which camera 15 typically is communicatively connected to a control unit 16 in the form of e.g. a microprocessor configured to control the camera 15. As is understood, the control unit 16 may be located remotely from the fridge 10 and be in communicative connection with the camera 15, e.g. via wireless communication.

[0039] Figure 4 shows a flowchart illustrating a method of detecting amount remaining of partly consumed food items located in a household appliance according to an embodiment for resolving the above-mentioned issue in the prior art.

[0040] In line with the above given example, assuming that a user opens a bottle 14, drinks some of the wine therein and then puts the bottle 14 back on one of the retainers 13. The camera 15 may thus be controlled to capture an image of the interior of the cabinet 11. For instance, this may occur by the control unit 16 being configured to instruct the camera 15 to capture an image each time the door 12 is closed and / or opened. Thus, in a first step S101, the control unit 16 acquires a visual representation of one or more of the bottles 14 in the cabinet 11 by controlling the camera 15 to capture one or more images of the interior of the cabinet 11.

[0041] In an example, upon the control unit 16 analysing the image captured during the previous closing with the image captured during the current closing, the control unit may determine that the specific bottle 14 was placed on the retainer 13 prior to the current closing of the door 12 and that further analysis of that bottle 14 is to be undertaken as will be described.

[0042] With reference to Figure 5, the control unit 16 may thus in an embodiment acquire a representation 19 a of the bottle 14 in the form of an image captured by the camera 15, which has the appearance of the bottle 14 lying down in the retainer 13 as shown in the upper illustration of Figure 5.

[0043] In an embodiment, it is envisaged that a projection transformation of the representation 19a of the bottle 14 is performed by the control unit 16 resulting in another representation 19b of the bottle 14 attaining a pose shown in the lower illustration of Figure 5. Since both the pose and position of the camera 15 and the pose and position of each bottle placed on the retainers are known, the projectiontransformation is performed to attain the pose of the bottle 14 (for each bottle being analysed, no matter which placement in the cabinet) shown in the lower illustration of Figure 5, which advantageously may facilitate further analysis as will be discussed. As is understood, projection transformation - i.e. to transform a pose of an object such that the transformed pose aligns with another given pose of the object - commonly also known as perspective-n-point (PnP) pose computation, is in itself known in the art and will not be described in further detail herein. In practice, a projection matrix maybe applied to first representation 19a for transforming the pose of the botte 14 to arrive at second representation 19b.

[0044] However, the projection transformation of Figure 5 is optional, and it may also be envisaged that the upper representation 19a with the bottle 14 lying down is used to determine bottle category and a level 20 of wine remaining in the bottle 14, as discussed below.

[0045] In a next step S102, the control unit 16 determines a particular category 21 of bottles to which the specific bottle 14 belongs, based on the upper or lower representation 19 a, 19b of the bottle 14 in Figure 5, for instance that the bottle is a smaller-size bottle (37.5 cl), a regular size (70 cl) or a magnum size (1.51) bottle and thereafter, as illustrated in Figure 6, to which category of wine bottles the specific wine bottle 14 belongs, e.g. Bordeaux, Burgundy, Alsace, Sparkling, etc. As is understood, the control unit 16 is preconfigured with these bottle types.

[0046] In this example, the control unit 16 compares in S102 the lower representation 19b of the bottle 14 of Figure 5 with preconfigured representations of bottles 2ia-2id illustrated in Figure 6 and finds a match with the left-most Bordeaux- type bottle 21a, thus concluding that the category of bottles to which the specific bottle 14 belongs is “Bordeaux”.

[0047] As is understood, with the projection transformation previously performed by the control unit 16, the pose of the representation 19b is aligned with the pose of the preconfigured bottles 2ia-2id with which the bottle 14 is compared for finding a match, which advantageously may facilitate the comparing.

[0048] Finally in S103, the control unit 16 determines, based on the captured image of the bottle 14, a detected level 20 of wine (see Figure 5) remaining in the bottle 14 as well as the determined bottle type (“Bordeaux”), which allows the controlunit 16 to estimate an amount of wine remining in the bottle 14, say 55 cl in this example. That is, for the particular level 20 and the known preconfigured bottle 21a, the control unit 16 computes the volume of wine remaining in the bottle 14.

[0049] As mentioned, it may be envisaged that no projection transformation is undertaken, and thus that the upper representation 19a with the bottle 14 lying down is used to determine the particular bottle category 21 and a level 20 of wine remaining in the bottle 14, by comparing to one or more known representations with bottles lying down.

[0050] With reference to Figure 7, the user may be alerted of the determined amount of wine remaining in the bottle 14, for instance via a display 17 on the cabinet 11 of the wine fridge 10, or via a smart phone 18 of the user. As is understood, the user maybe presented with a complete list of bottles 14 (opened or not) and their respective position on the retainers 13 via the display 17 and / or the smart phone 18. The smart phone may run a wine cabinet control app for this purpose and may communicate wirelessly with the control unit 16 using for instance Bluetooth, Wi-Fi, radio-frequency identification (RFID), near-field communication (NFC), etc. The wine cabinet control app may comprise virtual wine cellar functionality where the user records her wines upon wine purchase.

[0051] Advantageously, with this embodiment, the user is made aware of wine remaining in the opened bottle 14. In the art, as discussed, the user would herself need to keep track of opened bottles and the amount of wine remaining in each bottle.

[0052] In further embodiments, the user may be informed of the time instance when the bottled 14 was opened and even be provided with a recommended drinking window from the instance of opening the bottle 14. For instance, the wine app may be preconfigured with a drinking window of, say, 5 days, wherein the user is informed at one or more occasions after having opened the bottle 14 that the wine preferably should be drunk within the next 5 days.

[0053] As is understood, rather than using a built-in camera 15 of the wine fridge 10, the user may utilize the camera of her smart phone 18 to capture the image in step S101 from which the amount of wine remaining ultimately is determined in S102 andS103- If so, the embodiments above maybe realized even for a fridge 10 not being equipped with a camera 15.

[0054] Figure 8 illustrates a regular fridge 22 shown with doors 23, 24 opened and comprising various compartments 25, retainers 26, trays 27, bins 28 (possibly with transparent fonts 29), etc. As is understood, in such a fridge 22 there are numerus food items other than bottles for which it is possible to determine the amount remaining, such as meat, fish, number of vegetables remaining out of an original group of vegetables, e.g. tomatoes, cucumbers, onions, etc. In the following, an example will be discussed where it is assumed that the fridge is a regular fridge storing all kinds of food items in addition to bottles. As in the case of the previously illustrated wine fridge 10, the regular fridge 22 of Figure 8 comprises at least one camera 15 and control unit 16.

[0055] With reference to Figure 9, the upper illustration shows an image 30a of a food item in the form of a pizza 30 stored e.g. on a retainer 26 in the fridge 22, the image 30a being captured by the camera 15. As illustrated in the image 30a, one slice has been eaten from the pizza 30.

[0056] In an embodiment, the control unit 16 performs, after having acquired the image 30a from the camera 15, image segmentation on the image 30a to attain a representation 30b of the food item (i.e. the pizza 30) having higher contrast with respect to a background of the image, thereby resulting in the centre illustration of Figure 9 showing the image 30b after having been subjected to image segmentation. For instance, a binary mask maybe applied to the image, where “o ’’-valued pixels represent the background (i.e. black) while “i”-valued pixels represent the food item (i.e. white).

[0057] Thereafter, as previously has been described, the edge-detected representation 30b is transformed, the control unit 16 performs a projection transformation on the edge-detected image 30b resulting in a further representation 30c of the pizza 30c attaining a pose shown in the lower illustration of Figure 9, which pose thus is aligned with the pose of the preconfigured food items with which the pizza 30 is compared for finding a match.

[0058] In this example, the lower representation 30c is compared to preconfigured food items having a known weight, and ultimately there will be matchto a preconfigured representation of an (unconsumed) pizza, in which case the control unit 16 will conclude from the area of the pizza represented in image 30c and an image of an unconsumed pizza that there is a difference, and will thus be able to determine an amount remaining of the pizza 30, say 90% in this particular example (which maybe indicated as a weight rather than a percentage numeric).

[0059] In another embodiment, a captured image may indicate that there are, say, 5 onions in the fridge 10 as determined by the control unit 16, while there originally were 7 onions in the fridge 10 as indicated in a previously captured image (hence representing the preconfigured representation to which a comparison is made). The control unit 16 will thus as has been described above conclude that there are 5 remaining onions left out of the original 7 onions, and alert the user accordingly. As in the case of the pizza 30, it may also be envisaged that weight or volume of each individual onion is determined by comparing each onion to one or more known representations of an onion, the weight or volume of which is known.

[0060] Figure 10 illustrates a control device 16 configured to detect amount remaining of food items located in a household appliance according to an embodiment, where the steps of the method performed by the control device 16 in practice are performed by a processing unit 111 embodied in the form of one or more microprocessors arranged to execute a computer program 112 downloaded to a storage medium 113 associated with the microprocessor, such as a Random Access Memory (RAM), a Flash memory or a hard disk drive. The processing unit 111 is arranged to cause the control device 16 to carry out the method according to embodiments when the appropriate computer program 112 comprising computerexecutable instructions is downloaded to the storage medium 113 and executed by the processing unit 111. The storage medium 113 may also be a computer program product comprising the computer program 112. Alternatively, the computer program 112 maybe transferred to the storage medium 113 by means of a suitable computer program product, such as a Digital Versatile Disc (DVD) or a memory stick. As a further alternative, the computer program 112 maybe downloaded to the storage medium 113 over a network. The processing unit 111 may alternatively be embodied in the form of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc. The control device 16 further comprises a communicationinterface 114 (wired and / or wireless) over which the control device 16 is configured to transmit and receive data.

[0061] Figure 11 illustrates an embodiment for determining food items remaining using depth estimation of one or more images captured by the camera 15. Shown in Figure 11 are four bottles I4a-i4d. In an image containing multiple food items with different distances to the camera, further away items will look smaller than items closer to the camera 15. By estimating the depth from the camera 15 to the respective bottle I4a-i4d, it is possible to correctly estimate volume of the bottles. By just estimating the area of the bottles, the result may be that bottles 14a and i4d appear bigger, while bottles 14b and 14c appear smaller. For example, the bottles 14a and i4d may appear to be 11 bottles while bottles 14b and 14c may appear to be 0,75! bottles. By taking into account the depth information in the captured image, the control unit 16 may conclude that all bottles in fact have the same volume, e.g. being 11 bottles.

[0062] The aspects of the present disclosure have mainly been described above with reference to a few embodiments and examples thereof. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

[0063] Thus, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

CLAIMS1. A method for detecting an amount remaining of food items (14) located in a household appliance (10), comprising: acquiring (S101) a visual representation (19a) of at least one food item (14) located in the household appliance (10); determining (S102), from the acquired visual representation (19a), a category of preconfigured food items (2ia-2id) to which said at least one food item (14) belongs; determining (S103) an amount of said at least one food item (14) remaining with respect to a complete and non-consumed representation of the food item (14).

2. The method of claim 1, the acquiring (S101) of a visual representation of said at least one object (14) comprising: capturing an image of said at least one object (14) located in the household appliance (10).

3. The method of claims 1 or 2, the determining (S102) of a category (21) of food items to which said at least one food item (14) belongs comprising: comparing the acquired visual representation (19a) with preconfigured food items (2ia-2id), and if there is a match, the food item (14) of the acquired visual representation (19a) is considered to belong to a same category of food items as the preconfigured food item (21a) for which there is a match.

4. The method of any one of the preceding claims, wherein the determining (S103) of an amount of said at least one food item (14) remaining comprises, in case the food item (14) is represented by a container accommodating fluid: determining a level (20) of fluid remaining in the bottle; and estimating, from the determined level (20) and the determined category of containers (21a) to which the container belongs, an amount of fluid remaining in the container.

5. The method of any one of claims 1-3, wherein the determining (S103) of an amount of said at least one food item (14) remaining comprises: determining an area of the food item (30) in the acquired visual representation (30a); and estimating, from the determined area and a determined area of a food item ofthe determined category to which the food item belongs, an amount remaining of the food item.

6. The method of any one of the preceding claims, wherein the determining (S103) of an amount of said at least one food item (14) remaining comprises, in case a group of food items of the same category is determined to be present in the acquired visual representation, comprising: determining that the number of food items in the group is less than an originally registered number of food items for the group.

7. The method of any one of the preceding claims, further comprising: performing a projective transformation of a pose of the at least one food item (14) in the acquired visual representation (19a) to create a new representation (19b) of the at least one food item (14) having a pose being aligned with a pose of the preconfigured food items (2ia-2id) with which the food item (14) is compared for finding a match.

8. The method of any one of the preceding claims, further comprising: performing image segmentation on the acquired visual representation (30a) for creating a representation (30b) where the food item (30) has a higher contrast.

9. The method of claim 8, wherein the performing of image segmentation comprises applying a binary mask to the acquired the acquired visual representation (30a).

10. The method of any one of the preceding claims, further comprising: alerting the user of an estimated time instance when the food item should be consumed.

11. The method of any one of the preceding claims, further comprising: alerting the user of an exact location where the food item is located in the household appliance (10).

12. A device (16) configured to detect an amount remaining of food items (14) located in a household appliance (10), the device (16) being operative to: acquire (S101) a visual representation (19a) of at least one food item (14) located in the household appliance (10);determine (S102), from the acquired visual representation (19a), a category of preconfigured food items (2ia-2id) to which said at least one food item (14) belongs; determine (S103) an amount of said at least one food item (14) remaining with respect to a complete and non-consumed representation of the food item (14).

13. The device (16) of claim 12, being operative to, upon determining (S102) a category (21) of food items to which said at least one food item (14) belongs: compare the acquired visual representation (19a) with preconfigured food items (2ia-2id), and if there is a match, the food item (14) of the acquired visual representation (19a) is considered to belong to a same category of food items as the preconfigured food item (21a) for which there is a match.

14. The device (16) of any one of claims 12-13, being operative to, upon determining (S103) an amount of said at least one food item (14) remaining, in case the food item (14) is represented by a container accommodating fluid: determine a level (20) of fluid remaining in the bottle; and estimate, from the determined level (20) and the determined category of containers (21a) to which the container belongs, an amount of fluid remaining in the container.

15. The device (16) claims 12-14, being operative to, upon determining (S103) an amount of said at least one food item (14) remaining: determine an area of the food item (30) in the acquired visual representation (30a); and estimate, from the determined area and a determined area of a food item of the determined category to which the food item belongs, an amount remaining of the food item.

Images