Food indicator for refrigerators

Abstract

The present invention discloses embodiments for finding stored foods in refrigerators. When a user requests a food by speaking, the embodiment indicates location of the stored food by means of indicator lights in the refrigerator.

Description

FOOD INDICATOR FOR REFRIGERATORS

[0001] The present invention concerns a practical way of finding food stored in refrigerators.

[0002] Some refrigerators have inventory management units, which update inventory databases of stored foods by means of various techniques. For example, a refrigerator may have image capture devices to capture images when foods are put in or taken out of refrigerator and update the inventory by using image recognizing techniques. Another refrigerator may have a user interface display so that users may update the inventory by pressing related buttons on the display. The inventory management unit may also record ancillary information regarding stored foods. For example, the time that the food is put in or taken out or, identity of user who puts or takes the food or, expiration date of the food etc. may be logged.

[0003] Some inventory management units may also record storing locations of foods in refrigerators. Storing locations may be a specifically named compartment or a drawer or a basket or a shelf, or sub-locations within these locations such as front-left corner of basket, rear-right corner of shelf etc.

[0004] EP2719982B1 document mentions a user interface display on which user may touch related buttons to see stored foods on the display of a refrigerator. If a user needs to pick up a stored food, the user should first find that food on the display and understand where it is stored and then the user should open the refrigerator door to access that location to pick up the food. If the user needs to find another food, then the user should repeat the same process.

[0005] US11232329B2 document discloses an apparatus for providing user-customized information in a refrigerator. According to the disclosure, the apparatus recognizes the user and accordingly shows picture of the favorite food of the user on refrigerator display, and it informs the food location via an audio output device. Then the user should interpret the information declared by the audio output device or shown on the display and then understand the location of stored food and then open the door and access the food location. If the user needs to pick food other than the one that the apparatus suggested, then the user should open the door and seek it visually or by hand-check.

[0006] EP4018141A1 and US11920860B2 documents disclose a refrigerator which is capable to recognize name of a food from a user’s voice input to microphone and open door of related compartment based on the recognized food. Then the user should seek the compartment, visually and / or by hand, to access the food.Technical Problem

[0007] Most of the cases, refrigerators are full of mess of foods and users need to seek visually and by hand-checks to find specific foods as they need to pick. This situation is not only time-consuming but also it causes faster spoiling of foods stored in refrigerated cabinets due to longer opening duration of doors of the refrigerated cabinets.

[0008] Some refrigerators in prior art have some utilities for users to find stored foods as explained above. They need various user interactions to find stored foods and may not be practical for some users. Methods and apparatuses using user interface displays for finding stored foods require time-consuming human-machine interactions, and these systems use sophisticated technologies and are prone to make mistakes. Moreover, using human-machine interactions to understand stored food’s location virtually on a display is not a practical way for elderly users who are not familiar with user interface displays.Solution to Problem

[0009] The present invention discloses embodiments indicating storing locations and additional information related to a food in a refrigerator by means of indicator lights upon receiving a speech wherein a user says name of the food.Advantageous Effects of Invention

[0010] The objective of the present invention is to provide a practical way for immediately finding and physically seeing the location of the stored foods in a refrigerator, rather than virtually seeing or interpreting as in prior art.

[0011] is a block diagram illustrating control system with essential elements, in accordance with an embodiment of the present invention;

[0012] is a block diagram illustrating control system with additional elements, in accordance with an embodiment of the present invention;

[0013] is an exemplary content of inventory database of stored foods in a refrigerator, in accordance with an embodiment of the present invention;

[0014] is a perspective view illustrating the external appearance of a refrigerator in accordance with an embodiment of the present invention;

[0015] is a perspective view illustrating the internal appearance of the refrigerator in, in accordance with the embodiment of the present invention;

[0016] ,andillustrate an exemplary use of a refrigerator in accordance with an embodiment of the present invention;

[0017] andillustrate an example case of showing storing location of invisibly stored foods in a refrigerator in accordance with an embodiment of the present invention;

[0018] andillustrate an example case of indication additional and ancillary information related to stored foods in a refrigerator in accordance with an embodiment of the present invention;

[0019] andillustrate another example case of indication additional and ancillary information related to stored foods in a refrigerator in accordance with an embodiment of the present invention.

[0020] All the elements illustrated in figures are each assigned a reference numeral, and the corresponding terms of these numbers are listed as below:

[0021] 20 refrigerator

[0022] 21 body

[0023] 22 door

[0024] 41 control system with essential elements

[0025] 42 control systems with additional elements

[0026] 50 control unit

[0027] 51 inventory management unit

[0028] 52 inventory database

[0029] 52a essential data

[0030] 52b ancillary data

[0031] 53 audio input device

[0032] 54 speech recognition unit

[0033] 60 indicator light

[0034] 70 user interface display

[0035] 71 image capture device

[0036] 72 image recognition unit

[0037] 73 proximity sensor

[0038] 74 audio output device

[0039] 75 door sensor

[0040] 90 user

[0041] 91 requested food

[0042] 92 requesting keyword

[0043] Embodiments of the disclosure are described in detail with reference to the accompanying drawings. Identical or similar components are designated with the same reference numerals, and redundant descriptions are omitted. Known techniques are omitted where they obscure the disclosure's gist. The drawings illustrate embodiments and do not limit the technical scope, which encompasses all modifications, equivalents, and alternatives within the spirit and scope of the embodiments. Ordinal terms are used for distinction only. Singular forms include plural forms in detail descriptions and in appended claims, unless context dictates otherwise. It is to be understood that the device is not limited to the particular embodiments of the device described below, as variations of the particular embodiments may be made and still fall within the scope of the appended claims. It is also to be understood that the terminology employed is for the purpose of describing particular embodiments or aspects of embodiments and is not intended to be limiting. Instead, the scope of the present device will be established by the appended claims.

[0044] ‘Refrigerator’ phrase in this document describes a container used to store foods such as vegetables, fruits, meals, beverages, pantries which needs to be stored in a special climatic condition. According to that description, fridges, freezers, fridge-freezer combinations, beverage coolers, pantry coolers are all a kind of refrigerator 20.

[0045] The present invention discloses embodiments which indicate the location of a stored food in a refrigerator when a user says the name of the food as explained below.

[0046] Referring to, essential elements in block diagram for control system in any embodiment of the present invention, are refrigerator 20, inventory management unit 51, inventory database 52, audio input device 53, speech recognition unit 54, control unit 50, and indicator lights 60. These essential elements may be more than one in quantity and may be communicated by means of wired or wireless methods in embodiments of the present invention. Functions of these elements are described in detail as below:

[0047] The refrigerator 20, having a plurality of storage locations, stores foods.

[0048] The inventory management unit 51 updates the inventory database 52 by means of known methods in the art. The inventory database 52 maintains records of names of foods which are probable to be stored in refrigerator 20 and records of the stored foods in the refrigerator 20. The inventory database 52 may be recorded in memory chips as dedicated only to maintain the inventory database 52 or may be recorded in memory chips in another unit or may be recorded in an internet data server. One exemplary content of the inventory database 52 is illustrated inof an embodiment of the present invention. Essential data 52a required in an inventory database 52 are names of stored foods and their storing locations in the refrigerator 20. As shown in, various food names are already available in records in the inventory database 52, some foods are recorded as stored, some foods are not stored yet; the inventory management unit 51 updates the inventory database 52 when a stored food is taken out or when an food is put in. The inventory management unit 51 may also add more food names, by time, in the inventory database 52 by means of know methods in the art, such as machine learning and artificial intelligence. The inventory database 52 may also include ancillary data 52b beside essential data 52a; such as quantities, storing times, expiration times, storing users of the stored foods.

[0049] Referring to, the audio input device 53 may be a microphone or a device which receives audio and converts the audio to an analog electric signal and transmits the analog electric signal. The audio input device 53 may be arranged more than one in quantity to provide better reception of the speech from various positions of the user 90. The audio input device 53 receives speech from a user (90), wherein the user (90) says name of a requested food (91), and the audio input device 53 transmits the speech as analog signal to the speech recognition unit 54.

[0050] The speech recognition unit 54 converts the analog electric signal transmitted from the audio input device 53 to a digital signal, and processes the digital signal to generate text, and transmits the text to the control unit 50. The speech recognition unit 54 may process the digital signal to generate the text in its hardware or may share the digital signals to an internet server and receive back as the text. The speech recognition unit 54 may also be capable of recognizing more than one language. As a result, the audio input device 53 and the speech recognition unit 54 together provide conversion the speech of the user 90 to the text which is transmitted to the control unit 50 for triggering further processes. The speech recognition unit 54 may be also capable of identifying user 90 by analyzing the speech of the user 90, by means of known techniques in the art, therefore the speech recognition unit 54 may also transmit the user ID information to the control unit 50.

[0051] The control unit 50 interprets the text transmitted from the speech recognition unit 54 to determine requested food 91 of which name is said in the speech from the user 90, and then inquires storing locations of the requested food in the inventory database 52, and then controls illumination state of some indicator lights 60 to indicate inquired storing locations of the requested food 91. The control unit 50 may have dedicated hardware and software or may be integrated to another hardware and software in the refrigerator 20.

[0052] The indicator lights 60 may be any device of which illumination state can be changed by the control unit 50, such as LED (light-emitting diode) modules, LEP (light-emitting polymers) modules, LCD (liquid-crystal displays) modules, incandescent bulbs, neon bulbs, halogen bulbs, fluorescent bulbs, electroluminescent displays, electroluminescent lights or similar.

[0053] Referring to, there may be additional elements in block diagram for control system in another embodiment of the present invention to provide additional utilities. These additional elements are not limited to being one in quantity. These elements may be communicated by means of wired or wireless methods. Functions of these elements are described in detail as below:

[0054] The image capture device 71 may be a camera or a device which captures images and transmits the images to an image recognition unit 72.

[0055] The image recognition unit 72 recognizes content of the images transmitted from the image capture device 71, and identifies the users 90 and foods with their positions and motions in the recognized content, and transmits information including user ID, name of food, and other ancillary data to the control unit 50 or to the inventory management unit 51 or to the speech recognition unit 54, by means of known methods in the art. The inventory management unit 51 may update the inventory database 52 according to the information transmitted from the image recognition unit 72. The speech recognition unit 54 may use user ID information transmitted from the image recognition unit 72 during speech recognition process. Because image recognition processes and user ID registration processes are not in the scope of the present invention, they are not explained further.

[0056] The audio output device 74 may receive electric signals from the control unit 50 to announce ancillary information related to the requested food 91 as the indicator lights 60 indicate storing locations of the requested food 91 in an embodiment of the present invention. The audio output device 74 may be any device capable of converting electrical signals to sound which is hearable by the user 90. An example of announcing ancillary information is illustrated in.

[0057] The user interface display 70 may display ancillary information related to the requested food 91 as the indicator lights 60 indicate storing locations of the requested food 91 in an embodiment of the present invention. An example of displaying ancillary information is illustrated in.

[0058] The proximity sensor 73 may detect whether a user 90 is near the refrigerator 20, and inform the control unit 20 wherein the control unit 20 processes further steps accordingly. The proximity sensor 73 may be an infrared sensor or a device for detecting humans in vicinity of a pre-set zone.

[0059] A refrigerator 20, illustrated inand, in accordance with an embodiment of the present invention includes a body 21 forming a storage spaces, and 3 doors 22 forming closures to the body 21. In this embodiment the doors 22 also form storage spaces. However, another embodiment of the present invention may have any quantity of door 22 or may not have any door 22. Door 22 is not an essential element of the present invention.

[0060] The door sensor 73 may detect door state of a door 22 to inform whether the door 22 of the refrigerator 20 is open or closed, and transmit the door state information as ‘open’ or ‘close’ to the control unit 50 wherein the control unit 20 processes further steps accordingly. The door sensor 73 may be a mechanical switch or a reed switch or a hall effect sensor or a device for detecting open-close state of the door 22.

[0061] There may be additional features which are applicable to the embodiments in the present inventions. Here is explained:

[0062] The audio input device 53 may receive speeches including food names during ordinary talks, without any intention of food request from a user, and then the indicator lights 60 may be illuminated unnecessarily due to false activation. To prevent such false activations, the below solutions are applicable to embodiments of the present invention.

[0063] One basic solution to prevent the false activation is that: The control unit 50 may be further configured to check availability of a requesting keyword 92 in the speech of the user 90. If the requesting keyword 92 is available then the control unit 50 processes further steps to indicate the storing locations of the requested food 91; if the requesting keyword 92 is not available then the control unit 50 ignores the speech of the user 90, and does not process further steps to indicate the storing locations of the requested food 91. If the requesting keyword 92 is set to an unusual word which is not used in daily speech, then the false activation is eliminated in most cases. As illustrated in, assuming that the requesting keyword 92 is set to “my cool”; the control unit 50 will process “my cool ice cream” saying and indicate storing location of ice cream. The requesting keyword 92 may be a customizable parameter which is allowed to be set by the user 90.

[0064] Another solution to prevent the false activation is that: The proximity sensor 73 may detect whether a user is near the refrigerator 20 or not. If the proximity sensor 73 detects a user 90 then the control unit 50 processes further steps to indicate the storing locations of the requested food 91; if the proximity sensor does not detect a user 90 then the control unit 50 ignores the speech of the user 90, and does not process further steps to indicate the storing locations of the requested food 91.

[0065] Another solution to prevent the false activation is that: The control unit 50 may compare the user ID information transmitted from the image recognition unit 72 and the user ID information transmitted from the speech recognition unit 54 to check whether they match or not. If both user IDs, received from the image recognition unit 72 and the speech recognition unit 50, do not match then the control unit 50 ignores the speech of the user 90, and does not process further steps to indicate the storing locations of the requested food 91.

[0066] The indicator lights 60 may be positioned anywhere visible to the user 90, on the refrigerator 20. Positions and quantities of the indicator lights 60 may be arranged to indicate storing locations of the requested food 91, in ergonomic manner for human perception in embodiments of the present invention. Some indicator lights 60 may be positioned to be visible when the doors 22 are closed, some indicator lights 60 may be positioned to be visible when the doors 22 are open.

[0067] The indicator lights 60 may be in various shapes. The shape of the indicator lights 60 may be an arrow shape to indicate direction to access the requested food 91. Exemplary positions and shapes of the indicator lights 60 are illustrated inandof an embodiment of the present invention.

[0068] Illumination state of the indicator lights 60 for indicating storing locations of the requested food 91 in the refrigerator 20 may be extended to a pre-set duration, for example 3 seconds which is enough for human perception, in an embodiment of the present invention.

[0069] In another embodiment of the present invention, illumination state of the indicator lights 60 for indicating storing locations of the requested food 91 in the refrigerator 20 may be extended till another food request from the user 90.

[0070] In another embodiment of the present invention, illumination state of the indicator lights 60 for indicating storing locations of the requested food 91 in the refrigerator 20 may be extended till control unit 50 is informed by the image recognition unit 72 that the user 91 accesses the requested food 91.

[0071] In another embodiment of the present invention, illumination state of the indicator lights 60 for indicating storing locations of the requested food 91 in the refrigerator 20 may be extended till the control unit 50 is informed by the proximity sensor 73 that there is no human near the refrigerator 20.

[0072] In another embodiment of the present invention, the indicator lights 60, which are not at visible positions as the door 22 is closed, may be kept off if the door 22 is ‘closed’.

[0073] Illumination state of the indicator lights 60 may be controlled in various modes such as on, off, blinking, pulsing, specific color, specific brightness etc. in various embodiments of the present invention. The main objective of the indicator light 60 in the present invention is to indicate storing locations of the requested food 91; however, modes of the indicator lights 60 may also be controlled to indicate additional information related to the requested food 91, such as unavailability, about to expire, and expired. Here are some of the modes that are explained as examples, however modes are not limited to these; any mode of illumination state of the indicator lights 60 to indicate storing locations and additional information related to the requested food 91 is within the scope of the present invention.

[0074] In an embodiment of the present invention, specific indicator lights 60 related to the storing locations of the requested food 91 may be turned on while all other indicator lights 60 are off. In another embodiment of the present invention, specific indicator lights 60 related to the storing locations of the requested food 91 may be turned off while all other indicator lights are on; this mode of illumination state of the indicator lights 60may be useful for overall illumination of the refrigerator 20.

[0075] In another embodiment, the control unit 50 may be configured to control indicator lights 60 which indicate the storing locations of the requested food 91 to be illuminated in a specific color, for example ‘green’, while all other indicator lights 60 are illuminated in another color, for example ‘white’.

[0076] In another embodiment of the present invention, the control unit 50 may be configured to check availability of the requested food 91 in stocks in the inventory database 52; and change the illumination state of the indicator lights 60 in blinking mode or in specific color, for example ‘yellow color’, for a pre-set duration to inform that the requested food 91 is not available, if the requested food 91 is not available in stocks in the inventory database 52. The specific color to inform that the requested food 91 is not available, may be a customizable setting which is allowed to be set by the user 90.

[0077] In another embodiment of the present invention, the control unit 50 may be configured to determine the time remaining to the expiration time of the requested food 91 in the inventory database 52; and change the illumination state of the indicator lights 60 which indicate the storing locations of the requested food 91, into specific color, for example ‘orange color’, for a pre-set duration to inform that the requested food 91 is about to expire, if the time remaining to the expiration time of the requested food 91 in the inventory database 52 is less than a pre-determined time, for example 2 days. The specific color to inform that the requested food 91 is about to expire, may be a customizable setting which is allowed to be set by the user 90. The pre-determined time to compare the time remaining to the expiration time of requested food 91, may be a customizable setting which is allowed to be set by the user 90, or may be set by inventory management unit 51 based on the food and recorded into inventory database 52 as ancillary data. One exemplary indication of additional and ancillary information in accordance with an embodiment of the present invention is illustrated inand explained below paragraphs in detail.

[0078] In another embodiment of the present invention, the control unit 50 may be configured to determine whether the expiration time of the requested food 91 in the inventory database 52 is over; and change the illumination state of the indicator lights 60 which indicate the storing locations of the requested food 91, into specific color, for example ‘red color’, for a pre-set duration to inform that the requested food 91 is expired, if the expiration time of the requested food 91 in the inventory database 52 is over. The specific color to inform that the requested food 91 is expired, may be a customizable setting which is allowed to be set by the user 90. One exemplary indication of additional and ancillary information in accordance with an embodiment of the present invention is illustrated inand explained below paragraphs in detail.

[0079] Some examples of use scenarios are explained below:

[0080] As illustrated in, if the requesting keyword 92 is set to “my cool”, when a user 90 says “my cool ice cream” in vicinity of the refrigerator 20, then indicator lights 60 are illuminated as shown into indicate the storing location of the requested food 91 in an embodiment of the present invention. Then the user 90 accesses the requested food 91 as illustrated in.

[0081] As illustrated in, when a user 90 says “butter” in vicinity of the refrigerator 20, then indicator light 60 are illuminated as shown into indicate the storing location of the requested food 91 in an embodiment of the present invention. The requested food 91 in this case is stored at rear, behind another food, and not visible to the user 90; however, illuminated indicator light 60 glows the storing location of the requested food 91 so that the user 90 can perceive the storing location easily.

[0082] There may be multiple storing locations of the requested food 91; in these cases, all storing locations may be indicated by indicator lights 60 in an embodiment of the present invention as illustrated in. When a user 90 says “sauce” in vicinity of the refrigerator 20 as shown in, then three indicator lights 60 are illuminated as shown into indicate the storing locations of the requested food 91. Illumination color of one indicator light 60 illustrated inmay be specially differentiated (for example as ‘orange color’) to inform that use-by date of indicated requested food 91 is about to expire. The other indicator lights 60 are illuminated in a color (for example ‘white color’) determined only for indicating storing locations of the requested food 91. By this way, indicator lights 60 may also provide additional information related to the requested food 91. The control unit 50 in some embodiments of the present invention may also be configured to provide ancillary information of the requested food 91, as the indicator lights 60 indicate the storing locations, by means of the audio output device 74; for example, “sauce on lower balcony right will expire soon, please be informed” as illustrated in.

[0083] There may be some stored foods which are not proper for consuming or using when their expiration dates are over. For example, if use-by date for a food is over, that food should not be consumed. An embodiment of the present invention may alert the user 90 in case the expiration date of the requested food is over; an exemplary use is illustrated in. When a user 90 says “egg” in vicinity of the refrigerator 20, then 2 indicator lights 60 are illuminated as shown into indicate the storing locations of the requested food 91. Illumination color of one indicator light 60 illustrated inmay be specially differentiated (for example as ‘red color’) to inform that use-by date of indicated requested food 91 is expired. The other indicator lights 60 are illuminated in a color (for example ‘white color’) determined only for indicating storing locations of the requested food 91. By this way, indicator lights 60 may also provide alerts related to the requested food 91. The control unit 50 in some embodiments of the present invention may also be configured to provide ancillary information of the requested food 91, as the indicator lights 60 indicate the storing locations, by means of the user interface display 70; for example, “ALERT! Eggs on upper balcony right have expired.” as illustrated in.

Claims

A refrigerator (20), having a plurality of storage locations to store foods, comprising:at least one inventory database (52) to maintain records of stored foods and their storing locations in the refrigerator (20);at least one inventory management unit (51) to update the inventory database (52);at least one audio input device (53) to receive speech from a user (90);at least one speech recognition unit (54) to generate text content of the speech received by the audio input device (53);at least one control unit (50) to determine the requested food (91) in the text content generated by the speech recognition unit (54),characterized in that the refrigerator (20) further comprises:indicator lights (60), to indicate storing locations of the requested food (91);wherein the control unit (50) inquires the storing locations of the requested food (91) in the inventory database (52), and controls illumination state of the indicator lights (60) to indicate storing locations of the requested food (91).The refrigerator (20) according to claim 1, wherein the control unit (50) is further configured to:check availability of a requesting keyword (92) in the speech of the user (90); andignore the speech of the user (90) and not process further steps, in case the requesting keyword (92) is not available.The refrigerator (20) according to claim 1, further comprising:a proximity sensor (73) to detect a human near the refrigerator (20);wherein the control unit (50) is further configured to:ignore the speech of the user (90) and not process further steps, in case the proximity sensor (73) does not detect the user (90).The refrigerator according to claim 1, further comprising:at least one image capture device (71) to capture images;at least one image recognition unit (72) to recognize content of the images captured by the image capture device (71), and to transmit user ID information in the recognized content to the control unit (50);wherein the control unit (50) is further configured to:compare user ID information received from the image recognition unit (72) and user ID information received from the speech recognition unit (54); andignore the speech of the user (90) and not process further steps, in case both user IDs do not match.The refrigerator (20) according to any of the preceding claims, further comprising:at least one audio output device (74) for announcing information;wherein the control unit (50) is further configured to:receive ancillary information from the inventory database (52); andcontrol the audio output device (74) to announce the ancillary information related to the requested food (91), as the indicator lights (60) indicate storing locations of the requested food (91).The refrigerator (20) according to any of the preceding claims, further comprising:at least one user interface display (70) for displaying information;wherein the control unit (50) is further configured to:receive ancillary information from the inventory database (52); andcontrol the user interface display (70) to display the ancillary information related to the requested food (91), as the indicator lights (60) indicate storing locations of the requested food (91).The refrigerator (20) according to any of the preceding claims, further comprising:at least one door sensor (75) to transmit door state information of the door (22) as ‘open’ or ‘close’ to the control unit (50);wherein the control unit (50) is further configured to:keep off the indicator lights (60), which are not at visible positions as the door (22) is closed, if door state of the door (22) is ‘closed’.The refrigerator (20) according to any of the preceding claims, wherein the control unit (50) is further configured to:check the availability of the requested food (91) in stocks in the inventory database (52); andchange the illumination state of the indicator lights 60 into blinking mode or into specific color for a pre-set duration to inform that the requested food (91) is not available,if the requested food (91) is not available in stocks in the inventory database (52).The refrigerator (20) according to any of the preceding claims, wherein the control unit (50) is further configured to:determine the time remaining to the expiration time of the requested food (91) in the inventory database (52); andchange the illumination state of the indicator lights 60 which indicate the storing locations of the requested food (91), into specific color for a pre-set duration to inform that the requested food (91) is about to expire,if the time remaining to the expiration time of the requested food (91) in the inventory database (52) is less than a pre-determined time.The refrigerator (20) according to any of the preceding claims, wherein the control unit (50) is further configured to:determine whether the expiration time of the requested food (91) in the inventory database (52) is over; andchange the illumination state of the indicator lights 60 which indicate the storing locations of the requested food (91), into specific color for a pre-set duration to inform that the safe consumption time of the requested food (91) is expired,if the expiration time of the requested food (91) in the inventory database (52) is over.

Images