Refrigerator and control method therefor

Abstract

A refrigerator includes: a body including a storage chamber; a door including a door bin; a camera, arrangeable in the body, to photograph an inside of the body and an inside of the door; a memory; and a processor which obtains an image of at least a portion of the inside of the body and at least a portion of the door that is captured through the camera based on a trigger signal, detects a food object in the image, tracks the food object and identifies whether the food object is put in or taken out of the refrigerator, acquires feature information corresponding to the food object by inputting the image to a trained neural network model based on the food object being identified as being put in, and matches the image and the acquired feature information and stores the image and the acquired feature information in a food database.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is a continuation application is a continuation application, under 35 U.S.C. § 111(a), of international application No. PCT / KR2024 / 011445, Aug. 2, 2024, which claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2023-0113995, filed Aug. 29, 2023, and Korean Patent Application No. 10-2023-0185901, filed Dec. 19, 2023, the disclosures of which are incorporated herein by reference in their entireties.TECHNICAL FIELD

[0002] The disclosure relates to a refrigerator and a control method therefor, and more particularly, to a refrigerator that can manage putting in and taking out of food objects by photographing the inside of the refrigerator, and a control method therefor.BACKGROUND ART

[0003] In general, a refrigerator is a home appliance that includes a storage chamber storing food, and a cold air supplying device that supplies cold air to the storage chamber, and can thus keep food fresh for a long period.

[0004] In particular, a recent refrigerator includes a camera in a body including a storage chamber, and can photograph the inside of the body. Also, the refrigerator obtains information on food currently stored in the refrigerator based on the photographed image, and provides the obtained information on the food to a user.

[0005] However, in a conventional case, a plurality of cameras were included for obtaining correct information on food. However, in a refrigerator, limitations existed on installation locations of cameras and the number of installations for reasons such as a location wherein overlapping of foods was minimized, a location that was not influenced by a temperature, etc., and there was also a problem that the material cost rose.

[0006] Accordingly, there is a request for search of a method by which putting in and taking out of food objects can be correctly managed by using a minimum number of cameras.DISCLOSURE OF INVENTIONSolution to Problem

[0007] A refrigerator according to an embodiment of the disclosure includes a body including a storage chamber; a door, which is rotatably coupled to the body to open and close the storage chamber, including a door bin; a camera, arrangeable in the body, configured to photograph an inside of the body and an inside of the door; a memory to store at least one instruction; and a processor configured to: based on detecting a trigger signal, obtain an image of at least a portion of the inside of the body and at least a portion of the door that is photographed through the camera while the camera is arranged in the body, detect a food object included in the obtained image, track the food object and identify whether the food object is put in or taken out of the refrigerator, based on identifying that the food object is put in the refrigerator, obtain feature information corresponding to the food object by inputting the image into a trained neural network model, and match the image and the obtained feature information, and store the image and the obtained feature information based on the match in a food database.

[0008] The the processor is configured to: obtain information associated with the food object by inputting the image into a trained second neural network model, and store the information associated with the food object in the food database together with the image and the obtained feature information, and the information associated with the food object comprises: at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

[0009] The processor is configured to: crop an area including the food object from the image, and obtain the feature information corresponding to the food object by inputting an area where the food object is included into the trained neural network model.

[0010] The obtained feature information is first feature information, and the processor is configured to: based on identifying that the food object is taken out, identify the second feature information which was obtained when the food object was put in in the food database, and based on the identifying the second feature information, delete the second feature information from the food database.

[0011] The processor is configured to: based on the second feature information not being identified, identify a candidate list corresponding to the food object, and based on identifying the candidate list, provide user interface to receive user's selection for deleting at least one candidate from the food database, wherein the candidate list comprises: at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

[0012] The processor is configured to: based on identifying that the food object was taken out and not is put in within a predetermined time, delete the second feature information from the food database.

[0013] The food object comprises a first food object and a second food object, and the processor is configured to: identify that the second food object is put in the refrigerator, based on identifying that the second food object is put in, identify whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and based on identifying that the first food object and the second food object are matched, determine occurrence of an event where the first food object is put in again, and store information on the event in the food database.

[0014] The processor is configured to: based on identifying that the first food object matching the second food object does not exist, store the image and the obtained feature information as new food object.

[0015] The processor is configured to: based on identifying that the food object is put in, identify whether the feature information corresponding to the food object is obtainable, based on identifying the feature information corresponding to the food object is unobtainable, obtain information associated with an area corresponding to the food object from the image, and match the image and the information associated with the area corresponding to the food object, and store the image and the information with the area in the food database.

[0016] The processor is configured to: detect a hand object from the image, obtain identification information associated with the detected hand object based on a plurality of pre-stored hand objects, and match the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database.

[0017] Meanwhile, a control method for a refrigerator, the control method comprising: based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body; detecting a food object included in the obtained image; tracking the food object and identifying whether the food object is put in or taken out of the refrigerator; based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.

[0018] The control method further comprises: obtaining information associated with the food object by inputting the image into a trained second neural network model, and the storing comprises: storing the information associated with the food in the food database together with the image and the obtained feature information, and the information associated with the food object comprises: at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

[0019] The control method comprises: cropping an area including the food object from the image, and the obtaining the feature information comprises: obtaining the feature information corresponding to the food object by inputting an area wherein the food object is included into the trained neural network model.

[0020] The obtained feature information is first feature information, and wherein the control method comprises: based on identifying that the food object is taken out, identifying the second feature information which was obtained when the food object was put in in the food database, and based on the identifying the second feature information, deleting the second feature information from the food database.

[0021] The control method comprises: based on the second feature information not being identified, identifying a candidate list corresponding to the food object, and based on identifying the candidate list, providing user interface to receive user's selection for deleting at least one candidate from the food database, wherein the candidate list comprises: at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

[0022] The control method comprises: based on identifying that the food object was taken out and not is put in within a predetermined time, deleting the second feature information from the food database

[0023] The food object comprises a first food object and a second food object, and the control method comprises: identifying that the second food object is put in the refrigerator, based on identifying that the second food object is put in, identifying whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and based on identifying that the first food object and the second food object are matched, determining occurrence of an event where the first food object is put in again, and storing information on the event in the food database.

[0024] The control method comprises, based on identifying that the first food object matching the second food object does not exist, storing the image and the obtained feature information as new food object.

[0025] The control method comprises, based on identifying that the food object is put in, identifying whether the feature information corresponding to the food object is obtainable, based on identifying the feature information corresponding to the food object is unobtainable, obtaining information associated with an area corresponding to the food object from the image, and matching the image and the information associated with the area corresponding to the food object, and storing the image and the information with the area in the food database.

[0026] The control method comprises, detecting a hand object from the image, obtaining identification information associated with the detected hand object based on a plurality of pre-stored hand objects, and matching the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database.

[0027] Meanwhile, one or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of a refrigerator individually or collectively, cause the refrigerator to perform operations, the operations comprising: based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body; detecting a food object included in the obtained image; tracking the food object and identifying whether the food object is put in or taken out of the refrigerator; based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the disclosure;

[0029] FIG. 2A to FIG. 2E are diagrams for illustrating an inner configuration of a refrigerator according to an embodiment of the disclosure;

[0030] FIG. 3 is a flow chart for illustrating a method of managing putting in and taking out of food according to an embodiment of the disclosure;

[0031] FIG. 4 is a diagram for illustrating a method of tracking a food object and a hand object according to an embodiment of the disclosure;

[0032] FIG. 5 is a diagram illustrating a food list according to an embodiment of the disclosure;

[0033] FIG. 6 is a diagram illustrating a UI for correcting food included in a food list according to an embodiment of the disclosure;

[0034] FIG. 7 and FIG. 8 are diagrams for illustrating a food history list according to an embodiment of the disclosure;

[0035] FIG. 9 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0036] FIG. 10 is a diagram for illustrating an embodiment wherein food is put in again within a predetermined time according to an embodiment of the disclosure;

[0037] FIG. 11 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0038] FIG. 12 is a diagram for illustrating an embodiment of managing unspecified packaged objects according to an embodiment of the disclosure;

[0039] FIG. 13 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0040] FIG. 14 is a flow chart for illustrating a method of managing putting in and taking out of bundled food according to an embodiment of the disclosure;

[0041] FIG. 15 is a diagram for illustrating an embodiment of managing putting in and taking out of bundled food according to an embodiment of the disclosure;

[0042] FIG. 16 to FIG. 17B are diagrams for illustrating a method of managing putting in of food according to whether specific information was obtained according to an embodiment of the disclosure;

[0043] FIG. 18 is a flow chart for illustrating a method of managing putting in of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0044] FIG. 19 is a flow chart for illustrating a method of managing taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0045] FIG. 20 is a diagram for illustrating an embodiment of managing putting in and taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0046] FIG. 21 is a diagram for illustrating an object recognition model according to an embodiment of the disclosure;

[0047] FIG. 22 is a flow chart for illustrating a method of managing putting in of food according to whether food was normally put in according to an embodiment of the disclosure;

[0048] FIG. 23 is a flow chart for illustrating a method of managing taking out of food according to whether food was normally taken out according to an embodiment of the disclosure;

[0049] FIG. 24A and FIG. 24B are diagrams illustrating a UI providing information on food that is put in or taken out according to an embodiment of the disclosure;

[0050] FIG. 25A to FIG. 25C are diagrams for illustrating a candidate list for food that was taken out according to an embodiment of the disclosure; and

[0051] FIG. 26A and FIG. 26B are diagrams for illustrating an embodiment wherein a user corrects misrecognition of food that was taken out according to an embodiment of the disclosure.MODE FOR INVENTION

[0052] Various embodiments of the disclosure and the terms used in the embodiments are not for limiting the technological characteristics described in the disclosure to specific embodiments, but they should be interpreted to include various modifications, equivalents, or alternatives of the embodiments.

[0053] Also, with respect to the detailed description of the drawings, similar or related components may be designated by similar reference numerals.

[0054] In addition, a singular form of a noun corresponding to an item may include one of the item or a plurality of the items, unless instructed obviously differently in the related context.

[0055] Further, in the disclosure, each of the phrases “A or B,”“at least one of A and B,”“at least one of A or B,”“A, B, or C,”“at least one of A, B, and C,” and “at least one of A, B, or C” and the like may include any one of the items listed together with the phrase among the phrases, or all possible combinations of the listed items.

[0056] Also, the term “and / or” includes a combination of a plurality of related components described, or any one component among the plurality of related components described.

[0057] In addition, terms such as “first,”“second,” and the like may be used just to distinguish a component from another component, and are not intended to limit a component in another aspect (e.g.: importance or order).

[0058] Further, terms such as ‘the front surface,’‘the rear surface,’‘the top surface,’‘the bottom surface,’‘the side surface,’‘the left side,’‘the right side,’‘the upper part,’‘the lower part,’ etc. used in the disclosure were defined based on the drawings, and the shapes and locations of respective elements are not limited by these terms.

[0059] Also, terms such as “include” and “have” should be construed as designating that there are such characteristics, numbers, steps, operations, elements, components, or a combination thereof described in the disclosure, but not as excluding in advance the existence or possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components, or a combination thereof.

[0060] In addition, the description in the disclosure that one element is “connected with,”“combined with,”“supported by,” or “contacted with” another element not only includes a case wherein the elements are directly connected, combined, supported, or contacted, but also a case wherein the elements are indirectly connected, combined, supported, or contacted through a third element.

[0061] Further, the description in the disclosure that one element is “on top of” another element not only includes a case wherein the one element contacts the another element, but also a case wherein still another element exists between the two elements.

[0062] A refrigerator according to an embodiment may include a body.

[0063] “The body” may include an inner cabinet, an outer cabinet arranged on the outside of the inner cabinet, and a heat insulating material provided between the inner cabinet and the outer cabinet.

[0064] “The inner cabinet” may include at least one of a case forming a storage chamber, a plate, a panel, or a liner. The inner cabinet may be formed as one body, or may be formed as a plurality of plates are assembled. “The outer cabinet” may form the exterior of the body, and may be coupled to the outer side of the inner cabinet such that the heat insulating material is arranged between the inner cabinet and the outer cabinet.

[0065] “The heat insulating material” may insulate the inside of the storage chamber and the outside of the storage chamber such that the temperature inside the storage chamber can be maintained at a set appropriate temperature without being influenced by the outer environment of the storage chamber. According to an embodiment, the heat insulating material may include a foam heat insulating material. The foam heat insulating material may be molded by injecting and foaming urethane foam wherein polyurethane and a foaming agent are mixed between the inner cabinet and the outer cabinet.

[0066] According to an embodiment, the heat insulating material may additionally include a vacuum heat insulating material other than the foam heat insulating material, or the heat insulating material may consist only of a vacuum heat insulating material instead of the foam heat insulating material. The vacuum heat insulating material may include a core material, and an outer covering that houses the core material and seals the inside by vacuum or pressure close to vacuum. However, the heat insulating material is not limited to a foam heat insulating material or a vacuum heat insulating material as described above, and may include various materials that can be used for insulation.

[0067] “The storage chamber” may include a space limited by the inner cabinet. The storage chamber may further include an inner cabinet that limits the space corresponding to the storage chamber. In the storage chamber, various goods such as food, medicine, cosmetics, etc. may be stored, and the storage chamber may be formed such that at least one side is opened for putting in or taking out goods.

[0068] The refrigerator may include one or more storage chambers. When two or more storage chambers are formed in the refrigerator, each storage chamber may have different uses, and may be maintained at different temperatures. For this, each storage chamber may be partitioned from each other by a partition including a heat insulating material.

[0069] The storage chamber may be provided to be maintained in an appropriate temperature range according to uses, and may include “a refrigeration chamber,”“a freezer,” or “a temperature conversion chamber” that are divided according to the uses and / or the temperature ranges. The refrigeration chamber may be maintained at an appropriate temperature for keeping goods refrigerated, and the freezer may be maintained at an appropriate temperature for keeping goods frozen. “Refrigeration” may mean cooling goods to be cold within a limit of not freezing the goods, and as an example, the refrigeration chamber may be maintained in a range of 0° C. to 7° C. “Freezing” may mean cooling goods to be frozen or to be maintained in a frozen state, and as an example, the freezer may be maintained in a range of −20° C. to −1° C. The temperature conversion chamber may be used as any one of a refrigeration chamber or a freezer by the user's selection or regardless of it.

[0070] The storage chamber may be referred to by various names such as “a vegetable chamber,”“a fresh chamber,”“a cooling chamber,” and “an ice making chamber” other than the names such as “a refrigeration chamber,”“a freezer,” and “a temperature conversion chamber,” etc., and the terms such as “a refrigeration chamber,”“a freezer,” and “a temperature conversion chamber,” etc. used below should be understood as meaning that comprehensively includes storage chambers having uses and temperature ranges corresponding to each of them.

[0071] According to an embodiment, the refrigerator may include at least one door that is constituted to open and close the opened one side of the storage chamber. The door may be provided to open and close each of the one or more storage chambers, or may be provided such that one door opens and closes the plurality of storage chambers. The door may be rotatably or slidably installed on the front surface of the body.

[0072] “The door” may be constituted to seal the storage chamber when the door is closed. The door may include a heat insulating material like the body in order to insulate the storage chamber when the door is closed.

[0073] According to an embodiment, the door may include a door outer plate forming the front surface of the door, a door inner plate forming the rear surface of the door and facing the storage chamber, an upper cap, a lower cap, and a door heat insulating material provided in their insides.

[0074] On the rim of the door inner plate, a gasket that seals the storage chamber by being adhered to the front surface of the body when the door is closed may be provided. The door inner plate may include a dyke that projects toward the rear side such that a door bin that can keep goods is installed. Here, the door bin may be referred to as a door basket or a door bin.

[0075] According to an embodiment, the door may include a door body, and a front panel that is separably coupled to the front side of the door body and forms the front surface of the door. The door body may include a door outer plate forming the front surface of the door body, a door inner plate forming the rear surface of the door body and facing the storage chamber, an upper cap, a lower cap, and a door heat insulating material provided in their insides.

[0076] The refrigerator may be divided into a French door type, a side-by-side type, a bottom mounted freezer (BMF), a top mounted freezer (TMF), or a 1 door refrigerator, etc. according to the arrangement of the doors and the storage chambers.

[0077] According to an embodiment, the refrigerator may include a cold air supplying device that is provided to supply cold air to the storage chamber.

[0078] “The cold air supplying device” may include a machine, a tool, an electronic apparatus, and / or a system combining them that can cool the storage chamber by generating cold air and guiding the cold air.

[0079] According to an embodiment, the cold air supplying device may generate cold air through a freezing cycle including compression, condensation, expansion, and evaporation processes of a refrigerant. For this, the cold air supplying device may include a freezing cycle device including a compressor, a condenser, an expansion device, and an evaporator that can drive a freezing cycle. Also, according to an embodiment, the cold air supplying device may include a semiconductor such as a thermoelectric element. The thermoelectric element may cool the storage chamber with heat generating and cooling operations through a Peltier effect.

[0080] According to an embodiment, the refrigerator may include a machine chamber that is provided such that at least some components belonging to the cold air supplying device are arranged.

[0081] “The machine chamber” may be provided to be partitioned and insulated from the storage chamber for preventing transmission of heat generated from the components arranged in the machine chamber to the storage chamber. In order to radiate heat to the components arranged inside the machine chamber, the inside of the machine chamber may be constituted to communicate with the outside of the body.

[0082] According to an embodiment, the refrigerator may include a dispenser that is provided on the door to provide water and / or ice. The dispenser may be provided on the door such that a user can approach it without opening the door.

[0083] According to an embodiment, the refrigerator may include an ice making device that is provided to generate ice. The ice making device may include an ice making tray storing water, an ice moving device separating ice from the ice making tray, and an ice bucket storing ice generated in the ice making tray.

[0084] According to an embodiment, the refrigerator may include a controller for controlling the refrigerator.

[0085] “The controller” may include memory that stores or memorizes programs and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling the cold air supplying device, etc. according to the programs and / or the data memorized in the memory.

[0086] The memory stores or records various kinds of information, data, instructions, programs, etc. necessary for the operations of the refrigerator. The memory may memorize temporary data generated while a control signal for controlling the components included in the refrigerator is generated. The memory may include at least one of volatile memory or non-volatile memory or a combination of them.

[0087] The processor controls the overall operations of the refrigerator. The processor may control the components of the refrigerator by executing the programs stored in the memory. The processor may include a separate NPU that performs operations of an artificial intelligence model. Also, the processor may include a central processing unit, a graphic-dedicated processor (GPU), etc. The processor may generate a control signal for controlling the operations of the cold air supplying device. For example, the processor may receive temperature information of the storage chamber from the temperature sensor, and generate a cooling control signal for controlling the operations of the cold air supplying device based on the temperature information of the storage chamber.

[0088] Also, the processor may process a user input of a user interface according to the programs and / or the data memorized / stored in the memory, and control the operation of the user interface. The user interface may be provided by using an input interface and an output interface. The processor may receive a user input from the user interface. Also, the processor may transmit a display control signal for displaying an image on the user interface in response to a user input and image data to the user interface.

[0089] The processor and the memory may be provided integrally, or provided separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub processor. The memory may include one or more memories.

[0090] According to an embodiment, the refrigerator may include a processor and memory controlling all of the components included in the refrigerator, and include a plurality of processors and a plurality of memories individually controlling the components of the refrigerator. For example, the refrigerator may include a processor and memory controlling the operation of the cold air supplying device according to an output of the temperature sensor. Also, the refrigerator may separately include a processor and memory controlling the operation of the user interface according to a user input.

[0091] A communication module may communicate with external devices such as a server, a mobile device, another home appliance, etc. through an ambient access point (AP). The access point (AP) may connect a local area network (LAN) to which the refrigerator or a user device is connected to a wide area network (WAN) to which a server is connected. The refrigerator or the user device may be connected to the server through the wide area network (WAN).

[0092] An input interface may include a key, a touch screen, a microphone, etc. The input interface may receive a user input, and transmit it to the processor.

[0093] An output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated in the processor.

[0094] Meanwhile, in the disclosure, the meaning of the feature that the refrigerator 100“provides” information may not only include the feature of displaying information through a display 123 included in the refrigerator 100, but also the feature of transmitting information to a user terminal communicatively connected with the refrigerator 100 and displaying information through a display of the user terminal.

[0095] Hereinafter, the refrigerator according to various embodiments will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the disclosure. As illustrated in FIG. 1, the refrigerator 100 may include a camera 110, an outputter 120, a communication interface 130, a microphone 140, a sensor 150, memory 160, and a processor 170. The refrigerator 100 may be a device that is for keeping food or medicine, etc. at a predetermined temperature such that they are kept cool or do not rot. The refrigerator 100 according to an embodiment of the disclosure was illustrated in a form of a general household refrigerator, but is not limited thereto, and it may be a kimchi refrigerator, a liquor refrigerator, a cosmetic refrigerator, a freezer, etc.

[0096] The camera 110 is a component for generating a photographed image by photographing a subject, and here, a photographed image may include both of a moving image and a still image. Meanwhile, “an image” in the disclosure may be a concept including both of an image output on the display 123 and an image frame photographed by the camera 110. Also, in a photographed image, at least one object may be included. “An object” according to an embodiment of the disclosure is a component included in an image photographed through the camera, and may include a food object, a hand object, a face object, a kitchenware object, etc.

[0097] In particular, the camera 110 may photograph the storage chamber inside the body 230 of the refrigerator 100, and the door bin (or the door basket, the pantry) areas of the doors 210, 220. The camera 110 may be provided in at least one of the upper area, the lower area, or the side surface area inside the body 230 for photographing the inside of the body 230 and the door bin areas of the doors 210, 220. Also, the camera 110 may be provided outside the refrigerator 100 and photograph the outside of the refrigerator 100. In other words, the camera 110 can obviously be implemented not only as one camera, but also as a plurality of cameras depending on embodiments. More detailed explanation in this regard will be described with reference to FIG. 2A to FIG. 2E.

[0098] FIG. 2A is a diagram illustrating a refrigerator when the doors are opened according to an embodiment of the disclosure. The refrigerator 100 in FIG. 2A was illustrated such that a plurality of doors 210, 220 are provided on both sides of the upper part, but is not limited thereto, and the refrigerator 100 may be implemented as a French door type, a side-by-side type, etc. according to the arrangement of the doors 210, 220, the storage chamber 230, etc.

[0099] The refrigerator 100 may include doors 210, 220 on both sides of the upper part. Also, the refrigerator 100 may include a display 123 on one or more of the doors 210, 220 on both sides of the upper part. Here, on the display 123, not only information on the food stored in the refrigerator 100 but also various kinds of information (e.g., event information, notification information, recipe information, etc. received from the outside) may be provided.

[0100] The refrigerator 100 may further include a storage chamber 230 as illustrated in FIG. 2A. The storage chamber 230 may be opened by the doors 210, 220 that are opened and closed, and accommodate water, beverages, and refrigerated or frozen food. Here, the storage chamber 230 may include a plurality of accommodating spaces and storage spaces. The storage chamber 230 may be partitioned by partitions arranged inside the body. The storage chamber 230 may be divided into a freezing chamber arranged in the lower part of the refrigerator 100 and a refrigeration chamber arranged in the upper part. However, the arrangement of the freezing chamber and the refrigeration chamber is not limited thereto, and they may be arranged while their locations are changed with each other.

[0101] The doors 210, 220 may rotate by an angle set by a hinge (e.g., smaller than or equal to 300°), and open or close a portion of the front surface of the storage chamber 230.

[0102] Here, the second door 220 among the plurality of doors 210, 220 may include a display 123 that displays functions and the setting of the refrigerator 100 on the surface, and can be changed by a user's input (e.g., a touch or selection of a button). Other than the above, at least some doors among the plurality of doors 210, 220 may further include a dispenser that provides water, ice, or sparkling water and / or a handle that can be gripped, etc.

[0103] Meanwhile, as illustrated in FIG. 2A, a camera 110 may be included in the upper area of the body for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220.

[0104] FIG. 2B is an upper perspective view of the refrigerator 100, and as illustrated in FIG. 2B, the camera 110 may be arranged in the upper central area of the body (in particular, the top table) for photographing both of the door bins of the first door 210 and the second door 220. FIG. 2C is a cross-sectional view that cut the drawing in FIG. 2B by A-A′, and as illustrated in FIG. 2C, the camera 110 may be arranged to be toward a lower direction by a predetermined angle (e.g., 30 degrees) for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220.

[0105] FIG. 2D and FIG. 2E are upper perspective views of the refrigerator 100, and are diagrams illustrating photographing ranges 240-1, 240-2 photographed by the camera 110. FIG. 2D is a diagram illustrating the photographing range 240-1 photographed by the camera 110 while the doors 210, 220 of the refrigerator 100 are fully opened, and FIG. 2E is a diagram illustrating the photographing range 240-2 photographed by the camera 110 while the doors 210, 220 of the refrigerator 100 satisfied a predetermined condition. Here “the predetermined condition” is a condition for obtaining an optimal door bin image, and here, “an optimal door bin image” may be an image that satisfies an optimal condition for identifying information on the food included in the door bins.

[0106] Also, the camera 110 may provide a photographed image to the processor 170 for managing putting in and taking out of food.

[0107] In addition, the camera 110 may be implemented as a wide angle camera for photographing a wide field of view, but is not limited thereto.

[0108] Meanwhile, in FIG. 2A to FIG. 2E, it was explained that the camera 110 is located in the upper part inside the body, but this is merely an example, and it is obvious that a plurality of cameras can be provided in other areas (e.g., a rear surface area, a lower area, a side surface area, etc.) inside the body.

[0109] The outputter 120 may provide various feedbacks. In particular, as illustrated in FIG. 1, the outputter 120 may include a speaker 121, light emitting diodes (LED) 122, a display 123, etc., but this is merely an example, and the outputter 120 may further include other outputters (e.g., a haptic provision device, etc.).

[0110] Here, the speaker 121 may be provided inside or outside the refrigerator 100, and provide various auditory feedbacks through audio. The LED 122 may be provided inside the storage chamber 230 or the doors 210, 220 inside the refrigerator 100, and provide various visual feedbacks through an indicator in a specific form (e.g., an arrow, etc.) and flickering, etc. The display 123 may be located in at least some areas of the plurality of doors 210, 220, and provide various visual feedbacks to the user.

[0111] In particular, the outputter 120 may output information guiding that food was put in(or stored into) or taken out(or removed from) of the refrigerator 100, information on the food that was put in or taken out, a food list, a candidate list, etc. Detailed explanation in this regard will be described with reference to the drawings later.

[0112] The communication interface 130 may perform communication with an external server or an external terminal device. In particular, the communication interface 130 may transmit an image including food to an external server for obtaining information on the food, and receive information on the food from the external server. Also, the communication interface 130 may transmit information on food and information on a storage location of food, etc. to a user terminal, and receive a control command from the user terminal. Here, the communication interface 130 may directly perform communication with the user terminal, but this is merely an example, and the communication interface 130 can obviously perform communication with an external user terminal through a server.

[0113] In particular, the communication interface 130 may perform communication with various types of external devices by using various wireless communication technologies or mobile communication technologies. As such wireless communication technologies, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB) communication, Zigbee, infrared Data Association (IrDA), or near field communication (NFC), etc. may be included, and as mobile communication technologies, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, etc. may be included.

[0114] The microphone 140 is a component that obtains an audio signal and converts it into an electric signal, and may be provided inside or outside the refrigerator 100. In particular, the microphone 140 may receive an audio signal including a user voice. Here, the user voice may include information on putting in or taking out (referred to as “putting in / taking out” hereinafter) and information on food (e.g., types of food, expiry dates of food, etc.).

[0115] The sensor 150 may detect an operation state (e.g.: power or a temperature) of the refrigerator 100, or an external environmental state (e.g.: a user state), and generate an electric signal or a data value corresponding to the detected state. In particular, the processor 170 may measure each of the temperatures of the plurality of storage chambers of the refrigerator 100 through sensing values obtained through the sensor 150. Alternatively, the processor 170 may recognize that the user approached through the sensor 150, and may control the camera 110 to be in a prepared state. Alternatively, the processor 170 may detect opening of the door through the sensor 150, and drive the camera 110.

[0116] The memory 160 may store an operating system (OS) for controlling overall operations of the components of the refrigerator 100, and instructions or data related to the components of the refrigerator 100. In particular, the memory 160 may store various components for managing putting in / taking out of food. Also, the memory 160 may store a food database (DB) that stores information on the food stored in the refrigerator 100 (e.g., types of food, the capacity(or net contents) of food, expiry dates of food, storage locations of food, etc.).

[0117] Also, according to an embodiment, the memory 160 may store a trained first neural network model (e.g., an object classification model, etc.) for obtaining feature information corresponding to food that was put in or taken out of the refrigerator 100. Alternatively, the memory 160 may store a trained second neural network model (e.g., an object recognition model, etc.) for recognizing food that was put in or taken out of the refrigerator 100.

[0118] Meanwhile, the memory 160 may be implemented as non-volatile memory (ex: a hard disc, a solid state drive (SSD), flash memory), volatile memory (the memory inside the processor 170 may be included), etc.

[0119] The processor 170 may control the refrigerator 100 according to the at least one instruction stored in the memory 160.

[0120] In particular, the processor 170 may include one or more processors. Specifically, the one or more processors may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or a random combination of other components of the refrigerator, and perform operations regarding communication or data processing. The one or more processors may execute one or more programs or instructions stored in the memory. For example, the one or more processors may perform the method according to an embodiment of the disclosure by executing the one or more instructions stored in the memory.

[0121] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor, or performed by a plurality of processors. In other words, when a first operation, a second operation, and a third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first processor, or the first operation and the second operation may be performed by the first processor (e.g., a generic-purpose processor), and the third operation may be performed by a second processor (e.g., an artificial intelligence-dedicated processor). For example, an operation for managing putting in / taking out of food may be performed through a generic-purpose processor such as a CPU, etc., and an operation of obtaining feature information corresponding to a food object or an operation for recognizing food by using a neural network model may be performed by an artificial intelligence-dedicated processor such as an NPU, etc.

[0122] The one or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including a plurality of cores (e.g., multicores of the same kind or multicores of different kinds). In case the one or more processors are implemented as multicore processors, each of the plurality of cores included in the multicore processors may include internal memory of the processor such as cache memory, on-chip memory, etc., and a common cache shared by the plurality of cores may be included in the multicore processors. Also, each of the plurality of cores (or some of the plurality of cores) included in the multicore processors may independently read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction, or the plurality of entire cores (or some of the cores) may be linked with one another, and read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction.

[0123] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in the multicore processors, or they may be implemented by the plurality of cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first core included in the multicore processors, or the first operation and the second operation may be performed by the first core included in the multicore processors, and the third operation may be performed by a second core included in the multicore processors.

[0124] In the embodiments of the disclosure, the processor may mean a system on chip (SoC) wherein one or more processors and other electronic components are integrated, a single core processor, a multicore processor, or a core included in the single core processor or the multicore processor. Also, here, the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, etc., but the embodiments of the disclosure are not limited thereto.

[0125] According to an embodiment of the disclosure, if a trigger signal is detected, the processor 170 obtains an image by photographing at least a portion of the inside of the body and the doors 210, 220 through the camera 110. Here, the trigger signal is a signal for initiating a photographing operation of the camera 110, and may be generated by a user approach, opening of the doors, a user voice, etc. The processor 170 detects a first food object included in the obtained image. The processor 170 tracks the first food object and identifies whether the first food object is put in or taken out of the refrigerator. Here, “putting in” may be an operation of keeping food in the refrigerator, and “taking out” may be an operation of taking out food kept in the refrigerator. If it is identified that the first food object is put in, the processor 170 obtains first feature information corresponding to the first food object by inputting the image into the trained first neural network model. Here, the first neural network model may be an object classification model for obtaining feature information corresponding to food objects. The processor 170 matches the image and the obtained first feature information, and stores them in the food database.

[0126] According to an embodiment, the processor 170 may obtain information on the first food object by inputting the image into the trained second neural network model. Here, the second neural network model may be an object recognition model for recognizing food objects. The processor 170 may store the information on the food in the food database together with the image and the obtained first feature information. Here, the information on the first food object may include at least one of a type of the first food object, a product name of the first food object, the manufacturer of the first food object, or the capacity(or net contents) of the first food object.

[0127] According to an embodiment, the processor 170 may crop an area including the first food object from the image. Then, the processor 170 may obtain the first feature information corresponding to the first food object by inputting an area wherein the first food object is included into the trained first neural network model.

[0128] According to an embodiment, if it is identified that the first food object is taken out, the processor 170 may obtain second feature information corresponding to the first food object by inputting the image into the trained first neural network model. The processor 170 may identify whether food matched with the second feature information exists among the food stored in the food database. If it is identified that food matched with the second feature information exists, the processor 170 may delete the food matched with the second feature information from the food database.

[0129] According to an embodiment, the processor 170 may identify whether a second food object was put in within a predetermined time after it was identified that the first food object was taken out. If it is identified that the second food object was put in within the predetermined time, the processor 170 may identify whether the first food object and the second food object are matched. If it is identified that the first food object and the second food object are matched, the processor 170 may identify an event wherein the first food object is put in again, and store information related to the event wherein the first food object is put in again in the food database.

[0130] According to an embodiment, if it is identified that the second food object was not put in within the predetermined time or the first food object and the second food object are not matched, the processor 170 may identify whether food corresponding to the first food object exists among the food stored in the food database. If it is identified that food corresponding to the first food object exists, the processor 170 may delete the food corresponding to the first food object from the food database.

[0131] According to an embodiment, if it is identified that food corresponding to the first food object does not exist, the processor 170 may provide a candidate list for the first food object. Here, the candidate list may include at least one of n food objects corresponding to the most similar feature information to third feature information corresponding to the first food object, unspecified packaged objects, or food objects that were recently put in.

[0132] According to an embodiment, if it is identified that food corresponding to the first food object does not exist, the processor 170 may identify whether an unspecified packaged object that was put in before the predetermined time exists. If it is identified that the unspecified packaged object exists, the processor 170 may update the food database by determining that the first food object and the unspecified packaged object correspond.

[0133] According to an embodiment, if it is identified that the first food object is put in, the processor 170 may identify whether it is possible to obtain the first feature information corresponding to the first food object. If it is identified that it is impossible to obtain the first feature information corresponding to the first food object, the processor 170 may obtain information on an area corresponding to the first food object from the image. The processor 170 may match the image and the information on the area corresponding to the first food object, and store them in the food database.

[0134] According to an embodiment, the processor 170 may detect a hand object from the image. The processor 170 may obtain identification information on the detected hand object based on a plurality of pre-stored hand objects. The processor 170 may match the identification information on the hand object, the image, and the first feature information, and store them in the food database.

[0135] FIG. 3 is a flow chart for illustrating a method of managing putting in and taking out of food according to an embodiment of the disclosure.

[0136] The refrigerator 100 may detect a trigger signal in the operation S305. Here, the trigger signal is a signal for driving the camera 110, and may be, for example, a trigger signal generated after opening of the door is detected by a door opening / closing sensor, a trigger signal generated after a user approach is detected by a proximity sensor, and a trigger signal generated after a user input for driving the camera is input through the microphone 140. However, this is merely an example, and a trigger signal can obviously be generated by other methods.

[0137] The refrigerator 100 may obtain an image by driving the camera 110 in the operation S310. Driving the camera 110 is an operation of initiating an operation of the camera 110 so as to obtain an image, and may be expressed as activating the camera 110, or initiating the camera 110. Here, the refrigerator 100 may obtain an image based on initial set values of the camera (an exposure time, frames per second (FPS), a gain, etc.). Also, the refrigerator 100 may set a region of interest (ROI) area in the image. Here, the ROI area may be an area wherein an appearance that the user's hand or food is put in or taken out is identified.

[0138] Also, in case the camera 110 is located inside, regular illumination and a regular environment can be maintained according to the illumination inside the refrigerator 100, but in case the camera 110 is located outside, the photographing environment may change according to the outside environment (e.g., outside illumination, etc.). Accordingly, the refrigerator 100 may dynamically change the set values of the camera 110 based on the outside environment (e.g., the illumination direction, the illumination, etc.).

[0139] The refrigerator 100 may detect a food object in the operation S315. Here, the detected food object may be a moving food object (e.g., a food object gripped by the user's hand, a food object put in dishware such as a pot, a food object covered by another food object, etc.). Specifically, the refrigerator 100 may recognize a user operation by using a trained neural network model. Here, the trained neural network model is an operation recognition model trained to recognize hand operations of the user, and hand operations of the user can be learned by images photographed by various outside environments. The refrigerator 100 may detect a moving food object and a hand object by recognizing hand operations of the user.

[0140] According to an embodiment, the refrigerator 100 may crop an area including the detected food object from the photographed image. The refrigerator 100 may obtain the first feature information corresponding to the first food object by inputting the area wherein the first food object is included into the trained first neural network model. Alternatively, the refrigerator 100 may obtain information on the first food object by inputting the area wherein the first food object is included into the trained second neural network model.

[0141] Also, the refrigerator 100 may detect a pantry object, a face object, a kitchenware object, etc. other than a hand object and a food object.

[0142] The refrigerator 100 may track the food objects in the operation S320. Specifically, the refrigerator 100 may detect the food objects for each of the image frames obtained by the camera 110 and track the objects. Here, the refrigerator 100 may analyze the moving directions of the food objects based on the locations of the food objects included in the plurality of image frames.

[0143] According to an embodiment, as illustrated in FIG. 4, the refrigerator 100 may detect the moving directions of the food object 410 and the hand object 420 included in the plurality of image frames 400-1 to 400-6. Here, the refrigerator 100 may detect a center point of the food object 410 included in the image (in particular, the cropped area), and detect the moving direction of the food object 410 according to moving of the center point. Then, the refrigerator 100 may identify in which area of the refrigerator 100 the food object 410 was stored through the moving direction of the food object 410 detected in the first to fourth image frames 400-1 to 400-4. Then, as the food object 410 detected in the first to fourth image frames 400-1 to 400-4 was not detected in the fifth and sixth image frames 400-5, 400-6, the refrigerator 100 may determine an event wherein food is put in.

[0144] Meanwhile, in the aforementioned embodiment, it was explained that the food object 410 is detected and the food object is tracked, but this is merely an example, and the refrigerator 100 may track a pantry object, a face object, a kitchenware object, etc. other than a hand object and a food object.

[0145] The refrigerator 100 may determine whether a food object is put in or taken out in the operation S325. According to an embodiment, the refrigerator 100 may analyze a food object included in the plurality of image frames, and determine whether the food object is put in or taken out.

[0146] According to an embodiment, if an event wherein a food object was included in some image frames among the plurality of images frames, and then the food object is not included in the remaining image frames is detected, the refrigerator 100 may determine that the food object was put in. Alternatively, if an event wherein a food object was not included in some image frames among the plurality of images frames, and then the food object is included in the remaining image frames is detected, the refrigerator 100 may determine that the food object was taken out.

[0147] According to an embodiment, the refrigerator 100 may determine whether a food object was put in or taken out based on a moving direction of the food object included in the plurality of image frames. In other words, if it is identified that a food object included in the plurality of image frames moves to the direction of the inside of the body or the door bin, the refrigerator 100 may determine that the food object was put in. Alternatively, if it is identified that a food object included in the plurality of image frames moves to the direction of the outside of the refrigerator 100, the refrigerator 100 may determine that the food object was taken out.

[0148] Meanwhile, while detecting a food object and tracking it, the refrigerator 100 may obtain information on the food object by inputting a photographed image (in particular, an area corresponding to the cropped food object) into the trained second neural network model (an object recognition model). Here, the information on the food object may include at least one of the type of the food object, the product name of the food object, the manufacturer of the first food object, or the capacity of the first food object, but is not limited thereto.

[0149] If putting in of a food object is determined, the refrigerator 100 may input an image into the trained first neural network model in the operation S330. Here, the first neural network model may be a neural network model trained to obtain feature information corresponding to a food object by inputting an image. Here, the first neural network model may be trained by a contrastive learning method. The contrastive learning method is a main learning method of self-supervised learning, and is a method of learning such that feature information (or feature values) corresponding to similar images is located to be close to each other in a vector space, and feature information corresponding to different images is located to be far away from each other. By classifying objects through a neural network model trained by the contrastive learning method as above, even if parts of food objects are covered by a hand object, or food objects are photographed by different angles, or parts of food objects are deformed, or food objects are packaged by different packaging methods, correct classification of the objects can become possible by the first neural network model.

[0150] According to an embodiment, the first neural network model may subdivide a label unit and apply it to learning, and here, a label input by the user may be applied to learning of the first neural network model.

[0151] The refrigerator 100 may obtain first feature information from the first neural network model in the operation S335. Here, the first feature information may include a feature value corresponding to a food object. Here, the first feature information may be used for identifying food that will be taken out later.

[0152] The refrigerator 100 may match an image and the first feature information in the operation S340. In other words, the refrigerator 100 may match an image (in particular, an area corresponding to a food object) and the first feature information, and store them. Not only that, the refrigerator 100 may match information on a food object obtained by the second neural network model with an image and the first feature information, and store them. As an example, the refrigerator 100 may store information on a food object such as “the name of the food, the storage space of the food, the date of putting in of the food, the data of taking out of the food, the type of the food, barcode information of the food, a text included in the food,” etc. together with the image and the first feature information.

[0153] The refrigerator 100 may update the food DB in the operation S360. Specifically, the refrigerator 100 may match an image, the first feature information, and information on food with the food DB, and store them in the food DB. Here, the refrigerator 100 may store the first feature information in a feature information list of the food DB. In particular, the refrigerator 100 may update the feature information list by comparing similarity of the first feature information newly stored in the feature information list and other feature information, and aligning the first feature information through the comparison result. Not only that, the feature information list may align the feature information by considering user information such as a user pattern, time of putting in / taking out, etc. as well as the feature information.

[0154] If taking out of a food object is determined, the refrigerator 100 may input the image into the trained first neural network model in the operation S345. Here, as the first neural network model is identical to the first neural network model explained earlier, overlapping explanation will be omitted.

[0155] The refrigerator 100 may obtain second feature information from the first neural network model in the operation S350. Here, the second feature information may include a feature value corresponding to a food object. Also, although the second feature information was described as different information from the first feature information for distinguishing putting in / taking out, it may be the same value as the first feature information (or, a value within a range wherein it is determined as the same food type).

[0156] The refrigerator 100 may identify food matched with the second feature information in the operation S355. Specifically, the refrigerator 100 may identify food corresponding to the closest feature information to the second feature information among the feature information stored in the food DB stored in the refrigerator 100. Also, the refrigerator 100 may identify the food corresponding to the closest feature information to the second feature information as the food matched with the second feature information. In other words, as the feature information indicates a vector value within a vector space, the refrigerator 100 may calculate a distance between the second feature information and another feature information, and obtain information on the similarity of the second feature information and the another feature information.

[0157] Meanwhile, the refrigerator 100 may not only obtain food matched with the closest feature information to the second feature information, but also obtain N pieces of candidate feature information based on similarity between the second feature information and another feature information, and obtain a candidate list including the N pieces of candidate feature information. Here, N may be set in advance, but this is merely an example, and it may be changed by the user setting. The refrigerator 100 may receive input of a user command selecting food taken out to the user by using the candidate list.

[0158] The refrigerator 100 may update the food DB in the operation S360. Specifically, the refrigerator 100 may delete the food matched with the second feature information among the food stored in the food DB. Alternatively, the refrigerator 100 may change the information on the food matched with the second feature information among the food stored in the food DB. For example, the refrigerator 100 may change the number of the food stored in the food DB, etc.

[0159] The refrigerator 100 may provide a food list in the operation S365. Specifically, the refrigerator 100 may provide a food list based on the updated food DB. Here, the food list may include information on the food put into the plurality of areas of the refrigerator 100. In particular, the food list may include information on the food that was divided into each of the plurality of areas of the refrigerator 100 and stored in each of the plurality of areas. For example, the food list may divide information on the food stored in the left door bin, information on the food stored in the right door bin, information on the food stored in the first floor area inside the body, information on the food stored in the second floor area inside the body, and information on the food stored in the third floor area inside the body, and provide them to the user. Also, the food list may include information on the food that was put in, or information on the food that was taken out. Here, the information on the food that was put in or taken out may include information on the time of putting in or taking out, or the number of times of putting in or taking out, etc.

[0160] Meanwhile, according to the disclosure, the food DB may include a plurality of types of DBs. As an example, a first food DB may be a food DB including information on the food currently stored in the refrigerator 100. In other words, when the first food is put in, information on the first food may be added to the first food DB, and when the second food is taken out, information on the second food may be deleted from the first food DB. The first food DB may be used for managing the food currently stored in the refrigerator 100, and may also be used for providing a food list. As another example, a second food DB may be a food DB that temporarily stores information on the food currently taken out from the refrigerator 100 during a predetermined time. Here, the second food DB may be a food DB used for determining whether food taken out from the refrigerator 100 is put in again. As still another example, a third food DB may be a food DB storing information on the history of putting in / taking out of food. In other words, the third food DB may accumulate information on the history that food is put in, the history that food is taken out, etc. for each time zone, and store them. The third food DB may store all the histories of putting in / taking out of food, and may be used for the user to identify the history of putting in / taking out of food. Meanwhile, the first food DB may be referred to as “a food DB,” the second food DB may be referred to as “a temporarily stored food DB,” and the third food DB may be referred to as “a food history DB.”

[0161] Also, the refrigerator 100 may provide the history of taking out to the user by using a food history DB other than the food list. As an example, as illustrated in FIG. 5, the refrigerator 100 may provide a list 510 including food that was processed to have been taken out. Here, the list 510 including the food that was processed to have been taken out may be aligned for each type of the food as illustrated in FIG. 5, but this is merely an example, and the list 510 can obviously be aligned for each processing time of taking out.

[0162] The refrigerator 100 may update the food DB by using the list 510 including the food that was processed to have been taken out. Specifically, if a user command for selecting one food in the list 510 including the food that was processed to have been taken out which was illustrated in FIG. 5 and correcting the processing of taking out is input, the refrigerator 100 may provide a UI 610 for correcting the processing of taking out for the selected food. On the UI 610 for correcting the processing of taking out, a candidate list including a plurality of pieces of candidate food and a put back item may be provided. If one piece of candidate food is selected in the food list by the user, the refrigerator 100 may correct the processing of taking out for the food selected in the list 510, and perform the processing of taking out for the selected candidate food on the UI 610. By this, the refrigerator 100 may perform update such that the food selected in the list 510 is added to the food DB, and perform update such that the candidate food selected on the UI 610 is deleted from the food DB. Alternatively, if the “put back” item on the UI 610 for correcting the processing of taking out is selected, the refrigerator 100 may perform update such that the food selected in the list 510 is added to the food DB.

[0163] According to an embodiment of the disclosure, FIG. 7 and FIG. 8 are diagrams illustrating a food history list. Here, the food history list may be generated based on the food history DB.

[0164] The first list 710 illustrated in FIG. 7 may be a list that aligned information on food that was put in or taken out of the refrigerator 100 per date. Here, in the first list 710, the history of putting in / taking out of the food that was put in or taken out, information on the food that was put in or taken out, and images of the food that was put in or taken out may be included, as illustrated in FIG. 7. Also, the first list 710 may be aligned (or classified) by various standards. For example, the first list 710 may be aligned based on the dates of putting in / taking out by using the first UI element 720 in FIG. 7, and may be aligned by the names of the food and the categories of the food. Also, the first list 710 may be corrected according to a user input. For example, the refrigerator 100 may delete at least one selected among the food included in the first list 710 based on a user command input through the second UI element 730 in FIG. 7. Here, the first and second UI elements 720, 730 may be displayed on the first list 710.

[0165] Also, the first list 710 may be updated to include only the food that was put in or the food that was taken out. For example, if a user command for providing the second list including only the food that was put in (e.g., a user command selecting the “IN” item 715 illustrated in FIG. 7) is input, the refrigerator 100 may provide the second list 810 including only the food that was put in among the food included in the first list 710, as illustrated in FIG. 8.

[0166] Hereinafter, various embodiments of the disclosure will be explained with reference to the drawings. Meanwhile, as a method of identifying putting in or taking out of food through a photographed image was explained in detail in FIG. 3, overlapping explanation will be omitted in the following embodiments.

[0167] FIG. 9 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure.

[0168] The refrigerator 100 may identify taking out of food in the operation S905. Specifically, the refrigerator 100 may determine taking out of food detected from an image based on an image photographed by the camera 110, and obtain feature information corresponding to the detected food. Here, the refrigerator 100 may temporarily store information on the food that was taken out in a temporarily stored food DB. When a predetermined time passes, the refrigerator 100 may delete the information on the food that was taken out which was temporarily stored in the temporarily stored food DB.

[0169] The refrigerator 100 may identify whether putting in of food was generated within the predetermined time in the operation S910. In other words, the refrigerator 100 may identify whether putting in of food was generated within the predetermined time based on the time when taking out of food was identified (or the time of taking out). As an example, the refrigerator 100 may identify whether putting in of food was generated within the predetermined time based on the time of taking out for each of the plurality of pieces of food that were taken out. Here, the predetermined time may be a reference time for determining whether food was put in again, and may be, for example, one hour. Here, the predetermined time may be set according to a user input.

[0170] If it is identified that putting in of food was generated within the predetermined time in the operation S910-Y, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists in the operation S915. Here, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists based on feature information corresponding to the food that was put in within the predetermined time and feature information corresponding to the food that was taken out which is stored in the temporarily stored food DB. In other words, the refrigerator 100 may obtain (or calculate) the similarity (or the distance) between the feature information corresponding to the food that was put in within the predetermined time and the feature information corresponding to the food that was taken out which is stored in the temporarily stored food DB. Then, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists based on the obtained similarity (or distance).

[0171] If it is identified that food that was taken out which is matched with the food that was put in within the predetermined time exists in the operation S915-Y, the refrigerator 100 may determine that food was put in again in the operation S920. Here, putting in of food again means that food that was previously put in the refrigerator 100 is put in again within the predetermined time after being taken out, and it may be distinguished from the initial putting in.

[0172] The refrigerator 100 may update the food DB according to putting in of food again in the operation S925. Specifically, if it is determined that food was put in again, the refrigerator 100 may delete the information on the food that was put in again which is stored in the temporarily stored food DB, and store again the information on the food that was put in again in the food DB. Also, the refrigerator 100 may maintain the time of putting in of the food that was put in again as the initial time of putting in but not the time of putting in again, and update the food DB such that the history of putting in again (the time of taking out and the time of putting in again, etc.) is added.

[0173] If it is identified that putting in of food is not generated within the predetermined time in the operation S910-N, or if it is identified that food that was taken out which is matched with the food that was put in within the predetermined time does not exist in the operation S915-N, the refrigerator 100 may compare the food that was taken out and the food DB in the operation S930. In other words, the refrigerator 100 may obtain (or calculate) the similarity (or the distance) by comparing the feature information corresponding to the food that was taken out and the feature information of the food stored in the food DB.

[0174] The refrigerator 100 may identify whether food matched with the food that was taken out exists in the operation S935. Specifically, the refrigerator 100 may identify whether food having feature information of which similarity to the feature information corresponding to the food that was taken out is greater than or equal to a threshold value (or the distance is smaller than or equal to a threshold value) exists.

[0175] If it is identified that food matched with the food that was taken out exists in the operation S935-Y, the refrigerator 100 may determine taking out of food in the operation S940. In other words, the refrigerator 100 may determine taking out of food that was taken out in the operation S905.

[0176] The refrigerator 100 may update the food DB according to taking out of food in the operation S940. In other words, the refrigerator 100 may delete or change the information on the food matched with the food that was taken out among the food stored in the food DB.

[0177] If it is identified that food matched with the food that was taken out does not exist in the operation S935-N, the refrigerator 100 may provide a candidate list in the operation S950. Here, the candidate list is a list provided to the user for identifying food that was taken out, and may include at least one of n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in.

[0178] The refrigerator 100 may perform processing of taking out according to a user input in the operation S955. Specifically, if a user input selecting one of the plurality of pieces of food included in the candidate list is received, the refrigerator 100 may perform processing of taking out for the selected food (or unspecified packaged food).

[0179] The refrigerator 100 may update the food DB according to the processing of taking out in the operation S960. In other words, the refrigerator 100 may delete or change the information on the food matched with the food selected by the user among the food stored in the food DB.

[0180] According to an embodiment, if putting in of food is generated, the refrigerator 100 may determine whether putting in of the food is putting in of the food again by aligning a take-out list of food within the predetermined time from the time point of generation of putting in. Here, the take-out list of food may be a list generated based on the temporarily stored food DB. For example, the predetermined time may be one hour, but this is merely an example, and it may be changed according to a user input. Alternatively, the predetermined time may be changed according to a use pattern of the user. For example, the refrigerator 100 may obtain pattern information for the time when the same food object is put in after a food object is taken out, and set (or change) the predetermined time according to the pattern information. For example, if it is determined that the time pattern that the user puts in a food object again after taking out the food object is 10 minutes, the refrigerator 100 may set the predetermined time as 10 minutes.

[0181] FIG. 10 is a diagram for illustrating an embodiment wherein food is put in again within a predetermined time according to an embodiment of the disclosure. Meanwhile, each of the first to third images explained in FIG. 10 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0182] The refrigerator 100 may identify that milk was taken out at 15:10 on Aug. 10, 2023 through a first image 1010.

[0183] Then, the refrigerator 100 may obtain a second image 1020. Here, the refrigerator 100 may identify whether milk that was taken out was put in again within the predetermined time. In other words, the refrigerator 100 may identify that milk was put in through the second image 1020, and identify that the time when the milk was put in was 15:43 on Aug. 10, 2023 which is within the predetermined time (e.g., 1 hour) from the time point when the milk was taken out. Accordingly, the refrigerator 100 may identify that the milk was put in again through the second image 1020 and the time of putting in. Here, the refrigerator 100 may update the food DB according to putting in of the milk again. Specifically, the refrigerator 100 may delete the information on the milk that was put in again which was stored in the temporarily stored food DB, and store again the information on the milk that was put in again in the food DB. Also, the refrigerator 100 may maintain the time of putting in of the milk that was put in again as the initial time of putting in but not the time of putting in again, and update the food DB such that the history of putting in again (the time of taking out and the time of putting in again, etc.) is added.

[0184] Also, the refrigerator 100 may obtain a third image 1030. Here, the refrigerator 100 may identify whether milk that was taken out was put in again within the predetermined time. Here, the refrigerator 100 may identify that milk was put in through the third image 1030, and identify that the time when the milk was put in was 17:13 on Aug. 10, 2023 which passed the predetermined time (e.g., 1 hour) from the time point when the milk was taken out. Accordingly, the refrigerator 100 may identify that new milk was initially put in but not that milk was put in again through the third image 1030 and the time of putting in. Here, the refrigerator 100 may update the food DB according to putting in of new milk. Specifically, the refrigerator 100 may store information on the new milk and information on the initial time of putting in in the food DB.

[0185] By a method as above, the refrigerator 100 may manage the food DB by identifying putting in of food or putting in of food again, and store the history regarding putting in or putting in again in the food history DB.

[0186] FIG. 11 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure.

[0187] The refrigerator 100 may identify taking out of food in the operation S1110. Specifically, the refrigerator 100 may determine taking out of food detected from an image based on an image photographed by the camera 110, and obtain feature information corresponding to the detected food.

[0188] The refrigerator 100 may identify whether food that is matched exists by comparing the food that was taken out and the food DB in the operation S1120. Specifically, the refrigerator 100 may identify whether food having feature information of which similarity to the feature information corresponding to the food that was taken out is greater than or equal to a threshold value (or the distance is smaller than or equal to a threshold value) exists among the food stored in the food DB.

[0189] If it is identified that food matched with the food that was taken out exists in the operation S1120-Y, the refrigerator 100 may determine taking out of food in the operation S1130. Here, the refrigerator 100 may update the food DB according to taking out of food. In other words, the refrigerator 100 may delete or change the information on the food matched with the food that was taken out among the food stored in the food DB.

[0190] If it is identified that food matched with the food that was taken out does not exist in the operation S1120-N, the refrigerator 100 may provide a candidate list in the operation S1150. Here, the candidate list is a list provided to the user for identifying food that was taken out, and may include at least one of n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in. Here, the unspecified packaged food is food that is put in in an unspecified packaged state, and may be, for example, food that is put in in an unspecified packaged state of which content cannot be identified such as a black plastic bag, a shopping bag, a box, etc.

[0191] The refrigerator 100 may receive a user input selecting unspecified packaged food in the operation S1160. In other words, the refrigerator 100 may receive a user input selecting (or touching) unspecified packaged food among a plurality of pieces of food included in a displayed candidate list.

[0192] The refrigerator 100 may match the unspecified packaged food and the food that was taken out in the operation S1170. Specifically, the refrigerator 100 may match the information on the food that was taken out and information on the unspecified packaged food.

[0193] The refrigerator 100 may output an inquiry message to the user in the operation S1180. Specifically, the refrigerator 100 may output an inquiry message that checks whether the food that was taken out is matched with the unspecified packaged food to the user. For example, the refrigerator 100 may output an inquiry message which is “Is it correct that the apples that were taken out had been put in in a black plastic bag at 3:30 on August 10, 2023?” Here, the inquiry message may be provided through the display 123, but this is merely an example, and it may be provided through the speaker 121 in an auditory form. The refrigerator 100 may receive a user input for a response to the inquiry message.

[0194] The refrigerator 100 may update the food DB in the operation S1190. Specifically, if it is identified that the food that was taken out and the unspecific packaged food are matched (i.e., if a positive feedback input of the user for the inquiry message is received), the refrigerator 100 may delete or change the information on the unspecific packaged food stored in the food DB, and update the information on the unspecific packaged food stored in the food history DB to the information on the food that was taken out. If it is identified that the food that was taken out and the unspecific packaged food are not matched (i.e., if a negative feedback input of the user for the inquiry message is received), the refrigerator 100 may output the candidate list for inquiring about the food that was taken out again.

[0195] FIG. 12 is a diagram for illustrating an embodiment of managing unspecified packaged objects according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 12 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0196] The refrigerator 100 may identify that unspecified packaged food (e.g., food in a black plastic bag) was put in at 17:13 on Aug. 10, 2023 through a first image 1210. The refrigerator 100 may store information on the unspecified packaged food in the food DB.

[0197] Then, the refrigerator 100 may obtain a second image 1220. Here, the refrigerator 100 may identify that food that is taken out is apples through the second image 1220, and identify that apples were taken out at 17:15 on Aug. 10, 2023. The refrigerator 100 may identify whether apples exist in the food DB. If it is identified that apples do not exist in the food DB, the refrigerator 100 may provide a candidate list. The refrigerator 100 may identify that the apples correspond to the unspecified packaged food based on a user input that was input through the candidate list. The refrigerator 100 may update the unspecified packaged food stored in the food DB to apples, and update the food DB so as to remove the apples or reduce the number of the apples which are the unspecified packaged food stored in the food DB. Here, the refrigerator 100 may provide an inquiry message to the user for identifying the number of the apples stored in the unspecified packaged food. Alternatively, the refrigerator 100 may count the number of the apples without deleting the information on the unspecified packaged food in the food DB. Then, if it is identified that the unspecified packaged food (e.g., a black plastic bag) is taken out, the refrigerator 100 may delete the information on the unspecified packaged food in the food DB, and store the counted number of the apples in the food history DB.

[0198] FIG. 13 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure. Meanwhile, as the operations S1310 to S1330 disclosed in FIG. 13 correspond to the operations S1110 to S1130 disclosed in FIG. 11, overlapping explanation will be omitted.

[0199] The refrigerator 100 may identify whether unspecified packaged food that was recognized to have been put in before a predetermined time exists in the operation S1340. Here, the predetermined time may be a week, but is not limited thereto.

[0200] If it is identified that unspecified packaged food that was recognized to have been put in before the predetermined time exists in the operation S1340-Y, the refrigerator 100 may match the unspecified packaged food and food that was taken out in the operation S1350. Specifically, the refrigerator 100 may match information on the food that was taken out and the information on the unspecified packaged food with each other.

[0201] The refrigerator 100 may update the food DB in the operation S1360. Specifically, the refrigerator 100 may update the information on the unspecified packaged food stored in the food DB and the food history DB to the information on the food that was taken out.

[0202] The refrigerator 100 may perform processing of taking out of the food in the operation S1370. In other words, the refrigerator 100 may delete or change the information on the unspecified packaged food stored in the food DB.

[0203] If it is identified that unspecified packaged food that was recognized to have been put in before the predetermined time does not exist in the operation S1340-N, the refrigerator 100 may update the food DB in the operation S1380. Specifically, the refrigerator 100 may update information on taking out of the food that was taken out in the food DB. Alternatively, the refrigerator 100 may provide a candidate list for identifying the type of the food that was taken out.

[0204] FIG. 14 is a flow chart for illustrating a method of managing putting in and taking out of bundled food according to an embodiment of the disclosure.

[0205] The refrigerator 100 may detect putting in of bundled food A in the operation S1410. Here, bundled food may be food wherein a plurality of pieces of single food are stored in a form of a bundle. Here, the second neural network model may be trained by designating a label for bundled food at the time of training, and the refrigerator 100 may identify whether food is bundled food by using the second neural network model. In other words, the refrigerator 100 may obtain, as information on the bundled food A, information on whether it is bundled food, and information on the type of the bundled food, the number of pieces of food included in the bundled food, etc. Also, the refrigerator 100 may obtain feature information of the bundled food by using the first neural network model.

[0206] The refrigerator 100 may update the food DB in the operation S1420. In other words, the refrigerator 100 may store the information on the bundled food A in the food DB. In particular, the refrigerator 100 may store the information on the number of the pieces of single food corresponding to the bundled food A and the feature information of the bundled food A in the food DB.

[0207] The refrigerator 100 may detect taking out of the single food a in the operation S1430. In other words, the refrigerator 100 may detect taking out of the single food a constituting the bundled food A. In particular, the refrigerator 100 may obtain, as information on the single food a, information on the type of the single food and the number of the pieces of the single food that were taken out, etc.

[0208] The refrigerator 100 may obtain feature information of the single food a in the operation S1440. Here, the refrigerator 100 may obtain the feature information of the single food a by using the first neural network model. The first neural network model may be trained such that the feature information of the bundled food A and the feature information of the single food a correspond.

[0209] Then, the refrigerator 100 may match the bundled food A and the single food a based on the obtained feature information in the operation S1450. Specifically, the refrigerator 100 may identify that the bundled food A and the single food a are of the same type of food based on the feature information of the bundled food A and the feature information of the single food a, and match the bundled food A and the single food a based on the identification result.

[0210] The refrigerator 100 may update the food DB in the operation S1460. Here, the refrigerator 100 may update the food DB based on the number of the pieces of the single food a included in the bundled food A and the number of the pieces of the single food a that were taken out. In case the number of the pieces of the single food a included in the bundled food A and the number of the pieces of the single food a that were taken out are identical, the refrigerator 100 may delete the information on the bundled food A stored in the food DB. In case the number of the pieces of the single food a that were taken out is smaller than the number of the pieces of the single food a included in the bundled food A, the refrigerator 100 may change the number of the pieces of the single food a included in the bundled food A stored in the food DB.

[0211] FIG. 15 is a diagram for illustrating an embodiment of managing putting in and taking out of bundled food according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 15 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0212] The refrigerator 100 may identify that the bundled food A was put in at 17:00 on Mar. 24, 2023 through a first image 1510. The refrigerator 100 may obtain information on the bundled food A through the second neural network model. Here, the refrigerator 100 may store, as information on the bundled food A, the type of the bundled food (i.e., eggs) and the number of the pieces of the bundled food (e.g., seven) in the food DB. Also, the refrigerator 100 may obtain feature information for the bundled food A through the first neural network model.

[0213] Then, the refrigerator 100 may obtain a second image 1520. Here, the refrigerator 100 may identify food a that is taken out through the second image 1520, and identify that eggs were taken out at 18:40 on Mar. 24, 2023. The refrigerator 100 may obtain information on the single food a through the second neural network model. Here, the refrigerator 100 may obtain, as information on the single food a, the type of the single food (i.e., eggs) and the number of the pieces of the single food that were taken out (e.g., one). Also, the refrigerator 100 may obtain feature information for the single food a through the first neural network model. The refrigerator 100 may identify that the bundled food A and the single food a are matched food based on the feature information for the bundled food A and the feature information for the single food a, and change the number of the pieces of the bundled food stored in the food DB based on the identification result. In other words, the refrigerator 100 may change the number of the pieces of the bundled food stored in the food DB from seven to six. Accordingly, the refrigerator 100 can provide more correct information on the food inside the refrigerator 100 by integrally managing the bundled food A and the single food a.

[0214] FIG. 16 to FIG. 17B are flow charts for illustrating a method of managing putting in of food according to whether specific information was obtained according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 17B may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0215] The refrigerator 100 may determine putting in of food in the operation S1610. In other words, the refrigerator 100 may determine putting in of food based on a moving direction of a food object included in an image.

[0216] The refrigerator 100 may identify whether feature information for the food that was put in can be obtained in the operation S1620. Specifically, if an area of food gets to be covered by greater than or equal to a specific size by the user's hand while the food is being put in the refrigerator, the refrigerator 100 cannot obtain feature information corresponding to the food from an image. According to an embodiment, the refrigerator 100 may identify whether feature information for food that was put in can be obtained based on the size of the food included in an image. According to another embodiment, the refrigerator 100 may obtain feature information for food by inputting an image (or an area corresponding to the food) into the first neural network model. Then, the refrigerator 100 may identify whether food corresponding to the feature information exists (i.e., whether similar feature information exists). If it is identified that food corresponding to the feature information does not exist, the refrigerator 100 may identify that feature information for the food that was put in cannot be obtained.

[0217] If it is identified that feature information for the food that was put in can be obtained in the operation S1620-Y, the refrigerator 100 may match the image and the feature information for the food and store them in the food DB in the operation S1630, as explained above.

[0218] If it is identified that feature information for the food that was put in cannot be obtained in the operation S1620-N, the refrigerator 100 may obtain information on an area corresponding to the food object in the operation S1640. Specifically, the refrigerator 100 may obtain, as information on an area corresponding to the food object, texture information or information wherein the area corresponding to the food object was segmented from the image. Specifically, as illustrated in FIG. 17A, in case an image 1710 was obtained, the refrigerator 100 may segment an area 1720 corresponding to a food object, and obtain information on an area corresponding to the food object.

[0219] The refrigerator 100 may match the image and the information on the area corresponding to the food object, and store them in the operation S1650.

[0220] Afterwards, if food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may identify the food that was taken out based on an area corresponding to the food object.

[0221] Specifically, as illustrated in FIG. 17B, the refrigerator 100 may obtain a first image 1710, and identify that food was put in at 17:13 on Aug. 10, 2023 through the first image 1710. However, the refrigerator 100 may identify that feature information for the food cannot be obtained as the food was covered by the user's hand. The refrigerator 100 may obtain information on the area corresponding to the food, and store it.

[0222] Then, the refrigerator 100 may obtain a second image 1720. Here, the refrigerator 100 may identify that food that is taken out is milk through the second image 1720, and identify that milk was taken out at 17:15 on Aug. 10, 2023. According to an embodiment, the refrigerator 100 may identify that the milk that was taken out is the food for which feature information could not be obtained which was put in earlier, based on the information on the area corresponding to the food (texture information or segmented image information). The refrigerator 100 may update the food DB by matching the food for which feature information could not be obtained and the food that was taken out (milk). According to another embodiment, if milk which is food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may generate a candidate list, as explained above. Here, the candidate list may include food for which feature information could not be obtained other than n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in. If food for which feature information could not be obtained (i.e., an area corresponding to the food object) is selected through the candidate list, the refrigerator 100 may update the food DB by matching the food for which feature information could not be obtained and the food that was taken out. According to another embodiment, if milk which is food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may ask a question to the user through the speaker 121. For example, the refrigerator 100 may output a question message which is “Is the food that was put in the refrigerator at 17:13 on Aug. 10, 2023 milk?”, and update the food DB by matching the food for which feature information could not be obtained and the food that was taken out through the user's feedback (e.g., “Yes”). Here, the refrigerator 100 may output the question message to the user based on the time of putting in of the food for which feature information could not be obtained and the time of taking out of the food that was taken out.

[0223] Also, the refrigerator 100 may train the first neural network model based on an image including an area corresponding to a food object and food that was taken out. In other words, the refrigerator 100 may train the first neural network model to obtain feature information corresponding to food that was taken out when an area corresponding to a food object is input into the first neural network model.

[0224] FIG. 18 is a flow chart for illustrating a method of managing putting in of food according to identification information corresponding to a hand object according to an embodiment of the disclosure.

[0225] Putting in of food may be generated in the operation S1810. Here, the refrigerator 100 may photograph an image regarding putting in of the food.

[0226] The refrigerator 100 may detect a hand object and a food object from the photographed image in the operation S1820.

[0227] The refrigerator 100 may separate an area corresponding to the detected hand object in the operation S1830. In other words, the refrigerator 100 may separate an area corresponding to the hand object from among the hand object and the food object detected from the photographed image.

[0228] The refrigerator 100 may obtain identification information corresponding to the hand object in the operation S1840. Specifically, the refrigerator 100 may obtain identification information corresponding to the hand object by identifying the shapes of the fingers, nails, the back of the hand, or the palm included in the hand object. Here, the identification information corresponding to the hand object may have been stored in advance. In other words, the refrigerator 100 may photograph hands for each user and register them, and store identification information corresponding to the hand objects in advance.

[0229] The refrigerator 100 may obtain feature information corresponding to the food object in the operation S1850. Specifically, the refrigerator 100 may obtain feature information corresponding to the food object by inputting the image into the first neural network model.

[0230] The refrigerator 100 may match the identification information corresponding to the hand object, the image, and the feature information corresponding to the food object, and store them in the food DB in the operation S1860. In other words, the refrigerator 100 may store together not only the image and the feature information corresponding to the food object, but also information on the user who put in the food.

[0231] FIG. 19 is a flow chart for illustrating a method of managing taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure.

[0232] Putting in of food may be generated in the operation S1910. Here, the refrigerator 100 may photograph an image regarding putting in of the food.

[0233] The refrigerator 100 may detect a hand object and a food object from the photographed image in the operation S1920.

[0234] The refrigerator 100 may separate an area corresponding to the detected hand object in the operation S1930. In other words, the refrigerator 100 may separate an area corresponding to the hand object from among the hand object and the food object detected from the photographed image.

[0235] The refrigerator 100 may obtain identification information corresponding to the hand object in the operation S1940. Specifically, the refrigerator 100 may obtain identification information corresponding to the hand object by identifying the shapes of the fingers, nails, the back of the hand, or the palm included in the hand object.

[0236] The refrigerator 100 may identify whether the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are matched in the operation S1950. In other words, the refrigerator 100 may determine whether the identification information corresponding to the hand object at the time of taking out of the food that is taken out and the identification information corresponding to the hand object at the time of putting in are identical.

[0237] If it is identified that the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are matched in the operation S1950-Y, the refrigerator 100 may store a single user putting in / taking out pattern in the operation S1960. Here, the single user putting in / taking out pattern may refer to a pattern by which one user puts in and takes out food.

[0238] Then, the refrigerator 100 may update the food DB in the operation S1990. In other words, the refrigerator 100 may remove food that was taken out from the food DB, and store identification information corresponding to a hand object related to putting in / taking out in the food history DB.

[0239] If it is identified that the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are not matched in the operation S1950-N, the refrigerator 100 may provide a notification message notifying taking out by another user in the operation S1970. Specifically, if taking out is performed by another user, the refrigerator 100 may notify taking out by the another user through the display 123 or a user terminal.

[0240] The refrigerator 100 may store multi user putting in / taking out patterns in the operation S1980. Here, the multi user putting in / taking out patterns may refer to patterns by which a plurality of users put in and take out food.

[0241] Then, the refrigerator 100 may update the food DB in the operation S1990. In other words, the refrigerator 100 may remove food that was taken out from the food DB, and store identification information corresponding to a hand object related to putting in / taking out in the food history DB.

[0242] FIG. 20 is a diagram for illustrating an embodiment of managing putting in and taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure. Meanwhile, each of the first to fifth images explained in FIG. 20 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0243] The refrigerator 100 may identify putting in / taking out of food based on the first to fifth images (2010 to 2050) as illustrated in FIG. 20. Specifically, the refrigerator 100 may identify that “milk was taken out by the user A at 15:10 on Aug. 10, 2023” through the first image 2010. The refrigerator 100 may identify that “snacks were taken out by the user B at 15:21 on Aug. 10, 2023” through the second image 2020. The refrigerator 100 may identify that “milk was put in by the user B at 15:33 on Aug. 10, 2023” through the third image 2030. The refrigerator 100 may identify that “juice was taken out by the user C at 16:10 on Aug. 10, 2023” through the fourth image 2040. The refrigerator 100 may identify that “juice was taken out by the user C at 16:13 on Aug. 10, 2023” through the fifth image 2050.

[0244] As described above, the refrigerator 100 may identify and manage the types of food that is put in or taken out through a hand object and information on a user who puts in or takes out food.

[0245] As an example, if the same food is put in after being taken out by the same user within a predetermined time (e.g., 1 hour), the refrigerator 100 may identify that the food was put in again. However, if the same food is put in after being taken out by a different user even within the predetermined time (e.g., 1 hour), the refrigerator 100 may identify that new food was put in.

[0246] As an example, in case a user who put in food and a user who took out the same food are different, the refrigerator 100 may provide a notification to the users. Here, the refrigerator 100 may provide the notification through the display 123 or a user terminal.

[0247] As an example, the refrigerator 100 may extract food pattern information and store it. Here, the food pattern information may include information and pattern information for food that is put in or taken out in a specific time zone. The refrigerator 100 may utilize the food pattern information in update of the food list. Also, the refrigerator 100 may provide user pattern information to the user by using the food pattern information, and provide customized recommendation services for each user based on the user pattern information.

[0248] As an example, the refrigerator 100 may provide a sharing function among users who use the refrigerator 100. For example, in case the user A puts milk in the refrigerator 100, the refrigerator 100 may share the information through a message application, or share the information on the display 123 of the refrigerator 100. Also, the refrigerator 100 may track the user who takes out the milk put in by the user A, and the time zone of taking out.

[0249] FIG. 21 is a diagram for illustrating an object recognition model according to an embodiment of the disclosure.

[0250] An object recognition model 2120 is a model for recognizing a type of a food object, and it may be referred to as a second neural network model. In particular, the object recognition model 2120 may include a plurality of layers for recognizing a type of a food object. If a photographed image 2110 is input into the object recognition model 2120, the object recognition model 2120 may output the type of the food object included in the photographed image 2110, and a probability value thereof. As an example, the object recognition model 2120 may output an output value which is “tomatoes 85%” (2130). However, this is merely an example, and the object recognition model 2120 may output probability values for each of a plurality of candidate objects, and the refrigerator 100 may identify a candidate object having the highest probability value among the output probability values as the type of the food object. Also, the object recognition model 2120 may divide information on a type of a food object according to a plurality of levels and output it. For example, the object recognition model 2120 may output “major category (fresh food)—subcategory (vegetables)—detailed food name (asparagus)” or “major category (processed food)—subcategory (canned food)—detailed food name (tuna can)” as an output value.

[0251] The object recognition model 2120 according to an embodiment of the disclosure may be implemented as various neural network models such as a convolutional neural network (CNN), a Transformer, etc.

[0252] FIG. 22 is a flow chart for illustrating a method of managing putting in of food according to whether food was normally put in according to an embodiment of the disclosure.

[0253] The refrigerator 100 may recognize putting in of food in the operation S2210. Here, recognizing putting in of food may mean recognizing an operation of a food object of moving in the inside direction of the refrigerator 100.

[0254] The refrigerator 100 may identify whether the food was normally put in in the operation S2220. As an example, if a photographed image is input into the second neural network model, in case the type of the food object included in the photographed image cannot be identified (or, in case the probabilities that the food object may belong to specific types are all smaller than or equal to a predetermined value (e.g., 50%)), the refrigerator 100 may identify that the food was put in abnormally.

[0255] In case it was identified that the food was normally put in in the operation S2220-Y, the refrigerator 100 may add the food to the food list and output a success message in the operation S2230. In other words, the refrigerator 100 may add the food that was put in to the food list, and output a success message guiding that putting in of the food was recognized. Here, in the success message, information on the food that was put in and information on putting in may be included.

[0256] In case it was identified that the food was put in abnormally in the operation S2220-N, the refrigerator 100 may output a failure message in the operation S2240. In other words, the refrigerator 100 may output a failure message guiding failure to recognize putting in of the food. Here, in the failure message, a message inquiring about information on the food that was put in may be included.

[0257] FIG. 23 is a flow chart for illustrating a method of managing taking out of food according to whether food was normally taken out according to an embodiment of the disclosure.

[0258] The refrigerator 100 may recognize taking out of food in the operation S2310. Here, recognizing taking out of food may mean recognizing an operation of a food object of moving in the outside direction of the refrigerator 100.

[0259] The refrigerator 100 may identify whether the food was normally taken out in the operation S2320. As an example, if a photographed image is input into the second neural network model, in case the type of the food object included in the image cannot be identified, the refrigerator 100 may identify that the food was taken out abnormally.

[0260] In case it was identified that the food was taken out abnormally in the operation S2320-N, the refrigerator 100 may output a failure message in the operation S2330. In other words, the refrigerator 100 may determine that the food was taken out abnormally, and output a failure message. Here, abnormal taking out of the food may be an event wherein the type of the food that was taken out cannot be recognized. Here, in the failure message, a message inquiring about information on the food that was taken out may be included.

[0261] In case it was identified that the food was normally taken out in the operation S2320-Y, the refrigerator 100 may identify whether food matched with the recognized food that was taken out exists in the food DB in the operation S2340.

[0262] If it is identified that food matched with the recognized food that was taken out exists in the food DB in the operation S2340-Y, the refrigerator 100 may update the food DB, and output a success message in the operation S2350. In other words, the refrigerator 100 may remove or change the information on the food that was taken out in the food DB, and output a success message guiding that taking out of the food was recognized. Here, in the success message, information on the food that was taken out and information on taking out may be included.

[0263] If it is identified that food matched with the recognized food that was taken out does not exist in the food DB in the operation S2340-N, the refrigerator 100 may output a failure message guiding failure to recognize normal taking out of the food in the operation S2360. Here, in the failure message, a message inquiring about information on the food that was taken out may be included.

[0264] FIG. 24A and FIG. 24B are diagrams illustrating a UI providing information on food that is put in or taken out according to an embodiment of the disclosure.

[0265] When food is put in or taken out, the refrigerator 100 may update the food DB by reflecting information on the food that is put in or taken out. Then, the refrigerator 100 may provide a food list and information on the food that was put in or taken out based on the updated food DB. Here, at least one of the food list or the information on the food that was put in or taken out may be output whenever the door is closed after being opened, but this is merely an example, and it may be output based on a user input (e.g., a user touch).

[0266] As an example, as illustrated in FIG. 24A, the refrigerator 100 may provide a pop-up screen 2410 including information on the food that was put in and the food that was taken out. Here, the information on the food that was put in and the food that was taken out may be output on the display 123 of the refrigerator 100, but this is merely an example, and the information may be put on a display of a user terminal. Meanwhile, the feature that the refrigerator 100 provides the information on the food that was put in and the food that was taken out as the pop-up screen 2410 is merely an example, and the information may be provided in various forms (e.g., an image, a list, etc.).

[0267] Also, the refrigerator 100 may provide a voice assistant notification through the speaker 121. Alternatively, the refrigerator 100 may be connected with an external device (e.g., an AI speaker, etc.), and provide information on the food that was put in or taken out. For example, the refrigerator 100 may output a voice message which is “Cabbages, cheese, milk, and raw chicken were put on the shelf of the refrigerator. 2 L of milk, eggs, wine, and oranges were taken out” through the speaker 121 or a speaker of an external device.

[0268] Also, the refrigerator 100 may provide information for outputting a notification message (or a push message) to a user terminal. Here, the refrigerator 100 may provide information for outputting a notification message to a plurality of registered user terminals.

[0269] In addition, the refrigerator 100 may provide a notification message by being linked with a display device (e.g., a TV, etc.) among devices connected in a home.

[0270] Also, the refrigerator 100 may obtain recipe information based on information on the food that was taken out. For example, the refrigerator 100 may obtain recipe information related to the food that was taken out among the recipe information stored in the memory 160 in advance, and obtain recipe information related to the food that was taken out from an external server. Then, the refrigerator 100 may transmit the obtained recipe information to a device related to cooking of the food that was taken out (e.g., an oven, a microwave oven, a hood) among the devices connected in a home or a user terminal. By this, the user can check recipe information related to the food that was taken out more conveniently. In addition, the refrigerator 100 may identify a storage location of the food that was put in, and provide a UI screen 2420 including an inside image regarding the inside of the refrigerator 100 and the storage location of the food that was put in, as illustrated in FIG. 24B.

[0271] Also, the refrigerator 100 may provide information on whether the camera is being driven and recognition of putting in / taking out of food, etc. to the user as sound effects. Here, the sound effects may be output through the speaker 121 of the refrigerator 100, but this is merely an example, and the sound effects may be output through a speaker of an external device connected with the refrigerator 100. By this, the refrigerator 100 can immediately notify the user of information on whether the camera 110 operated normally, whether putting in / taking out of food was normally recognized, or was recognized incorrectly or was not recognized, etc. through sound effects, even if the history of putting in / taking out is not output on the display screen all the time.

[0272] Also, the refrigerator 100 may provide a sound effect regarding normal driving of the camera as a first sound effect, provide a sound effect regarding normal recognition of putting in of food as a second sound effect, provide a sound effect regarding normal recognition of taking out of food as a third sound effect, provide a sound effect regarding an error in putting in of food (non-recognition or misrecognition of food) as a fourth sound effect, and provide a sound effect regarding an error in taking out of food (non-recognition or misrecognition of food) as a fifth sound effect. Here, the first to fifth sound effects may be different from one another, but this is merely an example, and the second and third sound effects may be identical, and the fourth and fifth sound effects may also be identical. The first sound effect may be output on a time point when the camera started to be driven normally after the door is opened. The second sound effect or the third sound effect may be output whenever one piece of food is normally put in or taken out. The fourth sound effect or the fifth sound effect may be output whenever the user's hand / food was detected, but the type of the food cannot be identified.

[0273] FIG. 25A to FIG. 25C are diagrams for illustrating a candidate list for food that was taken out according to an embodiment of the disclosure.

[0274] When taking out of food is determined, the refrigerator 100 may not only match the food to the most similar food stored in the food DB, but may also output a candidate list including food having similar feature information. Here, the number of the pieces of food included in the candidate list may be changed, and the food may be output as much as the number designated by the user.

[0275] In particular, when the operation of managing putting in or taking out of food ends, the refrigerator 100 may provide the information on the food that was put in or taken out in a form of a list to the user. Then, if one of the food that was put in or taken out included in the list is selected, the refrigerator 100 may provide a candidate list for changing the selected food to another food.

[0276] For example, the refrigerator 100 may output a list 2510 including information on the food that was put in or taken out as illustrated in FIG. 25A. Here, if the first food 2511 is selected among the food included in the list 2510, the refrigerator 100 may display a candidate list 2520 including n pieces of candidate food 2521, 2522, 2523 having similar feature information to the feature information of the first food 2511 as illustrated in FIG. 25B. Here, the selected first food 2511 may be displayed to be distinguished from other food. Then, as illustrated in FIG. 25B, if the second candidate food 2522 in the candidate list is selected, the refrigerator 100 may change the first food 2511 to the second candidate food 2522 and display it, as illustrated in FIG. 25C. Further, in the candidate list, new fourth candidate food 2524 may be added. By this, the user can change the type of the recognized food that was taken out to another type, and thus user convenience can be increased.

[0277] FIG. 26A and FIG. 26B are diagrams for illustrating an embodiment wherein a user corrects misrecognition of food that was taken out according to an embodiment of the disclosure.

[0278] The refrigerator 100 may output the history of food that was processed to have been taken out in the food DB per set period as a pop-up screen. In particular, the refrigerator 100 may store the history of the food that was processed to have been taken out in the food history DB, and output it per set period. Here, the set period may be, for example, a week, but this is merely an example, and it may be changed by a user input.

[0279] As an example, as illustrated in FIG. 26A, the refrigerator 100 may display a pop-up screen 2610 including information on the food that was processed to have been taken out.

[0280] When the pop-up screen 2610 is output, the user may check the history of the food that was processed to have been taken out, and if there is food that was not actually taken out, but was processed to have been taken out, the user may add the food to the food DB again. For example, if the first food (e.g., eggs) and the second food (e.g., carrots) are selected on the pop-up screen 2610, and then the “add to the food list” button is selected, the refrigerator 100 may add the first food and the second food to the food DB again, and display a guide message 2620 which is “Added to the food list,” as illustrated in FIG. 26B.

[0281] As an example, the refrigerator 100 may display a button for purchase of the food that was processed to have been taken out. When the button is selected, the refrigerator 100 may connect to a website for purchasing the food that was processed to have been taken out and display a food purchase page. By this, the user can purchase the food that was processed to have been taken out more conveniently.

[0282] Meanwhile, the orders in the follow charts in the aforementioned various embodiments are merely examples, and it is obvious that the orders of each operation in the flow charts can be changed, and the orders in the flow charts can also be performed simultaneously.

[0283] Meanwhile, the processor 170 according to an embodiment of the disclosure performs control to process input data according to predefined operation rules or an artificial intelligence model stored in the memory 160. The predefined operation rules or the artificial intelligence model are characterized in that they are made through learning.

[0284] Here, being made through learning means that a learning algorithm is applied to a plurality of training data, and predefined operation rules or an artificial intelligence model having desired characteristics are thereby made. Such learning may be performed in a device itself wherein artificial intelligence is performed according to the disclosure, or through a separate server / system.

[0285] An artificial intelligence model (e.g., first and second object detection networks) may consist of a plurality of neural network layers. The at least one layer has at least one weight value, and performs an operation of the layer through an operation result of the previous layer, and at least one defined operation. As examples of a neural network, there are a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann Machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a Transformer, but the neural network in the disclosure is not limited to the aforementioned examples excluding specified cases.

[0286] A learning algorithm is a method of training a specific subject device by using a plurality of training data and thereby making the specific subject device make a decision or make prediction by itself. As examples of learning algorithms, there are supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but learning algorithms in the disclosure are not limited to the aforementioned examples excluding specified cases.

[0287] Meanwhile, methods according to the various embodiments of the disclosure may be provided while being included in a computer program product. A computer program product refers to a product, and it can be traded between a seller and a buyer. A computer program product can be distributed in the form of a storage medium that is readable by machines (e.g.: compact disc read only memory (CD-ROM)), or distributed on-line (e.g.: download or upload) through an application store (e.g.: Play Store™), or directly between two user devices (e.g.: smartphones). In the case of on-line distribution, at least a portion of a computer program product (e.g.: a downloadable app) may be stored in a storage medium readable by machines such as the server of the manufacturer, the server of the application store, and the memory of the relay server at least temporarily, or may be generated temporarily.

[0288] Also, the method according to the various embodiments of the disclosure may be implemented as software including instructions stored in machine-readable storage media, which can be read by machines (e.g.: computers). The machines refer to apparatuses that call instructions stored in a storage medium, and can operate according to the called instructions, and the apparatuses may include an electronic apparatus according to the aforementioned embodiments (e.g.: a refrigerator).

[0289] Meanwhile, a storage medium that is readable by machines may be provided in the form of a non-transitory storage medium. Here, the term ‘a non-transitory storage medium’ only means that the device is a tangible device, and does not include a signal (e.g.: an electromagnetic wave), and the term does not distinguish a case wherein data is stored semi-permanently in a storage medium and a case wherein data is stored temporarily. For example, ‘a non-transitory storage medium’ may include a buffer wherein data is temporarily stored.

[0290] In case an instruction is executed by a processor, the processor may perform a function corresponding to the instruction by itself, or by using other components under its control. An instruction may include a code that is generated or executed by a compiler or an interpreter.

[0291] Also, while preferred embodiments of the disclosure have been shown and described, the disclosure is not limited to the aforementioned specific embodiments, and it is apparent that various modifications may be made by those having ordinary skill in the technical field to which the disclosure belongs, without departing from the gist of the disclosure as claimed by the appended claims. Further, it is intended that such modifications are not to be interpreted independently from the technical idea or prospect of the disclosure.

Examples

Embodiment Construction

[0052]Various embodiments of the disclosure and the terms used in the embodiments are not for limiting the technological characteristics described in the disclosure to specific embodiments, but they should be interpreted to include various modifications, equivalents, or alternatives of the embodiments.

[0053]Also, with respect to the detailed description of the drawings, similar or related components may be designated by similar reference numerals.

[0054]In addition, a singular form of a noun corresponding to an item may include one of the item or a plurality of the items, unless instructed obviously differently in the related context.

[0055]Further, in the disclosure, each of the phrases “A or B,”“at least one of A and B,”“at least one of A or B,”“A, B, or C,”“at least one of A, B, and C,” and “at least one of A, B, or C” and the like may include any one of the items listed together with the phrase among the phrases, or all possible combinations of the listed items.

[0056]Also, the ter...

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is a continuation application is a continuation application, under 35 U.S.C. § 111(a), of international application No. PCT / KR2024 / 011445, Aug. 2, 2024, which claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2023-0113995, filed Aug. 29, 2023, and Korean Patent Application No. 10-2023-0185901, filed Dec. 19, 2023, the disclosures of which are incorporated herein by reference in their entireties.TECHNICAL FIELD

[0002] The disclosure relates to a refrigerator and a control method therefor, and more particularly, to a refrigerator that can manage putting in and taking out of food objects by photographing the inside of the refrigerator, and a control method therefor.BACKGROUND ART

[0003] In general, a refrigerator is a home appliance that includes a storage chamber storing food, and a cold air supplying device that supplies cold air to the storage chamber, and can thus keep food fresh for a long period.

[0004] In particular, a recent refrigerator includes a camera in a body including a storage chamber, and can photograph the inside of the body. Also, the refrigerator obtains information on food currently stored in the refrigerator based on the photographed image, and provides the obtained information on the food to a user.

[0005] However, in a conventional case, a plurality of cameras were included for obtaining correct information on food. However, in a refrigerator, limitations existed on installation locations of cameras and the number of installations for reasons such as a location wherein overlapping of foods was minimized, a location that was not influenced by a temperature, etc., and there was also a problem that the material cost rose.

[0006] Accordingly, there is a request for search of a method by which putting in and taking out of food objects can be correctly managed by using a minimum number of cameras.DISCLOSURE OF INVENTIONSolution to Problem

[0007] A refrigerator according to an embodiment of the disclosure includes a body including a storage chamber; a door, which is rotatably coupled to the body to open and close the storage chamber, including a door bin; a camera, arrangeable in the body, configured to photograph an inside of the body and an inside of the door; a memory to store at least one instruction; and a processor configured to: based on detecting a trigger signal, obtain an image of at least a portion of the inside of the body and at least a portion of the door that is photographed through the camera while the camera is arranged in the body, detect a food object included in the obtained image, track the food object and identify whether the food object is put in or taken out of the refrigerator, based on identifying that the food object is put in the refrigerator, obtain feature information corresponding to the food object by inputting the image into a trained neural network model, and match the image and the obtained feature information, and store the image and the obtained feature information based on the match in a food database.

[0008] The the processor is configured to: obtain information associated with the food object by inputting the image into a trained second neural network model, and store the information associated with the food object in the food database together with the image and the obtained feature information, and the information associated with the food object comprises: at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

[0009] The processor is configured to: crop an area including the food object from the image, and obtain the feature information corresponding to the food object by inputting an area where the food object is included into the trained neural network model.

[0010] The obtained feature information is first feature information, and the processor is configured to: based on identifying that the food object is taken out, identify the second feature information which was obtained when the food object was put in in the food database, and based on the identifying the second feature information, delete the second feature information from the food database.

[0011] The processor is configured to: based on the second feature information not being identified, identify a candidate list corresponding to the food object, and based on identifying the candidate list, provide user interface to receive user's selection for deleting at least one candidate from the food database, wherein the candidate list comprises: at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

[0012] The processor is configured to: based on identifying that the food object was taken out and not is put in within a predetermined time, delete the second feature information from the food database.

[0013] The food object comprises a first food object and a second food object, and the processor is configured to: identify that the second food object is put in the refrigerator, based on identifying that the second food object is put in, identify whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and based on identifying that the first food object and the second food object are matched, determine occurrence of an event where the first food object is put in again, and store information on the event in the food database.

[0014] The processor is configured to: based on identifying that the first food object matching the second food object does not exist, store the image and the obtained feature information as new food object.

[0015] The processor is configured to: based on identifying that the food object is put in, identify whether the feature information corresponding to the food object is obtainable, based on identifying the feature information corresponding to the food object is unobtainable, obtain information associated with an area corresponding to the food object from the image, and match the image and the information associated with the area corresponding to the food object, and store the image and the information with the area in the food database.

[0016] The processor is configured to: detect a hand object from the image, obtain identification information associated with the detected hand object based on a plurality of pre-stored hand objects, and match the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database.

[0017] Meanwhile, a control method for a refrigerator, the control method comprising: based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body; detecting a food object included in the obtained image; tracking the food object and identifying whether the food object is put in or taken out of the refrigerator; based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.

[0018] The control method further comprises: obtaining information associated with the food object by inputting the image into a trained second neural network model, and the storing comprises: storing the information associated with the food in the food database together with the image and the obtained feature information, and the information associated with the food object comprises: at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

[0019] The control method comprises: cropping an area including the food object from the image, and the obtaining the feature information comprises: obtaining the feature information corresponding to the food object by inputting an area wherein the food object is included into the trained neural network model.

[0020] The obtained feature information is first feature information, and wherein the control method comprises: based on identifying that the food object is taken out, identifying the second feature information which was obtained when the food object was put in in the food database, and based on the identifying the second feature information, deleting the second feature information from the food database.

[0021] The control method comprises: based on the second feature information not being identified, identifying a candidate list corresponding to the food object, and based on identifying the candidate list, providing user interface to receive user's selection for deleting at least one candidate from the food database, wherein the candidate list comprises: at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

[0022] The control method comprises: based on identifying that the food object was taken out and not is put in within a predetermined time, deleting the second feature information from the food database

[0023] The food object comprises a first food object and a second food object, and the control method comprises: identifying that the second food object is put in the refrigerator, based on identifying that the second food object is put in, identifying whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and based on identifying that the first food object and the second food object are matched, determining occurrence of an event where the first food object is put in again, and storing information on the event in the food database.

[0024] The control method comprises, based on identifying that the first food object matching the second food object does not exist, storing the image and the obtained feature information as new food object.

[0025] The control method comprises, based on identifying that the food object is put in, identifying whether the feature information corresponding to the food object is obtainable, based on identifying the feature information corresponding to the food object is unobtainable, obtaining information associated with an area corresponding to the food object from the image, and matching the image and the information associated with the area corresponding to the food object, and storing the image and the information with the area in the food database.

[0026] The control method comprises, detecting a hand object from the image, obtaining identification information associated with the detected hand object based on a plurality of pre-stored hand objects, and matching the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database.

[0027] Meanwhile, one or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of a refrigerator individually or collectively, cause the refrigerator to perform operations, the operations comprising: based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body; detecting a food object included in the obtained image; tracking the food object and identifying whether the food object is put in or taken out of the refrigerator; based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the disclosure;

[0029] FIG. 2A to FIG. 2E are diagrams for illustrating an inner configuration of a refrigerator according to an embodiment of the disclosure;

[0030] FIG. 3 is a flow chart for illustrating a method of managing putting in and taking out of food according to an embodiment of the disclosure;

[0031] FIG. 4 is a diagram for illustrating a method of tracking a food object and a hand object according to an embodiment of the disclosure;

[0032] FIG. 5 is a diagram illustrating a food list according to an embodiment of the disclosure;

[0033] FIG. 6 is a diagram illustrating a UI for correcting food included in a food list according to an embodiment of the disclosure;

[0034] FIG. 7 and FIG. 8 are diagrams for illustrating a food history list according to an embodiment of the disclosure;

[0035] FIG. 9 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0036] FIG. 10 is a diagram for illustrating an embodiment wherein food is put in again within a predetermined time according to an embodiment of the disclosure;

[0037] FIG. 11 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0038] FIG. 12 is a diagram for illustrating an embodiment of managing unspecified packaged objects according to an embodiment of the disclosure;

[0039] FIG. 13 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure;

[0040] FIG. 14 is a flow chart for illustrating a method of managing putting in and taking out of bundled food according to an embodiment of the disclosure;

[0041] FIG. 15 is a diagram for illustrating an embodiment of managing putting in and taking out of bundled food according to an embodiment of the disclosure;

[0042] FIG. 16 to FIG. 17B are diagrams for illustrating a method of managing putting in of food according to whether specific information was obtained according to an embodiment of the disclosure;

[0043] FIG. 18 is a flow chart for illustrating a method of managing putting in of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0044] FIG. 19 is a flow chart for illustrating a method of managing taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0045] FIG. 20 is a diagram for illustrating an embodiment of managing putting in and taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure;

[0046] FIG. 21 is a diagram for illustrating an object recognition model according to an embodiment of the disclosure;

[0047] FIG. 22 is a flow chart for illustrating a method of managing putting in of food according to whether food was normally put in according to an embodiment of the disclosure;

[0048] FIG. 23 is a flow chart for illustrating a method of managing taking out of food according to whether food was normally taken out according to an embodiment of the disclosure;

[0049] FIG. 24A and FIG. 24B are diagrams illustrating a UI providing information on food that is put in or taken out according to an embodiment of the disclosure;

[0050] FIG. 25A to FIG. 25C are diagrams for illustrating a candidate list for food that was taken out according to an embodiment of the disclosure; and

[0051] FIG. 26A and FIG. 26B are diagrams for illustrating an embodiment wherein a user corrects misrecognition of food that was taken out according to an embodiment of the disclosure.MODE FOR INVENTION

[0052] Various embodiments of the disclosure and the terms used in the embodiments are not for limiting the technological characteristics described in the disclosure to specific embodiments, but they should be interpreted to include various modifications, equivalents, or alternatives of the embodiments.

[0053] Also, with respect to the detailed description of the drawings, similar or related components may be designated by similar reference numerals.

[0054] In addition, a singular form of a noun corresponding to an item may include one of the item or a plurality of the items, unless instructed obviously differently in the related context.

[0055] Further, in the disclosure, each of the phrases “A or B,”“at least one of A and B,”“at least one of A or B,”“A, B, or C,”“at least one of A, B, and C,” and “at least one of A, B, or C” and the like may include any one of the items listed together with the phrase among the phrases, or all possible combinations of the listed items.

[0056] Also, the term “and / or” includes a combination of a plurality of related components described, or any one component among the plurality of related components described.

[0057] In addition, terms such as “first,”“second,” and the like may be used just to distinguish a component from another component, and are not intended to limit a component in another aspect (e.g.: importance or order).

[0058] Further, terms such as ‘the front surface,’‘the rear surface,’‘the top surface,’‘the bottom surface,’‘the side surface,’‘the left side,’‘the right side,’‘the upper part,’‘the lower part,’ etc. used in the disclosure were defined based on the drawings, and the shapes and locations of respective elements are not limited by these terms.

[0059] Also, terms such as “include” and “have” should be construed as designating that there are such characteristics, numbers, steps, operations, elements, components, or a combination thereof described in the disclosure, but not as excluding in advance the existence or possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components, or a combination thereof.

[0060] In addition, the description in the disclosure that one element is “connected with,”“combined with,”“supported by,” or “contacted with” another element not only includes a case wherein the elements are directly connected, combined, supported, or contacted, but also a case wherein the elements are indirectly connected, combined, supported, or contacted through a third element.

[0061] Further, the description in the disclosure that one element is “on top of” another element not only includes a case wherein the one element contacts the another element, but also a case wherein still another element exists between the two elements.

[0062] A refrigerator according to an embodiment may include a body.

[0063] “The body” may include an inner cabinet, an outer cabinet arranged on the outside of the inner cabinet, and a heat insulating material provided between the inner cabinet and the outer cabinet.

[0064] “The inner cabinet” may include at least one of a case forming a storage chamber, a plate, a panel, or a liner. The inner cabinet may be formed as one body, or may be formed as a plurality of plates are assembled. “The outer cabinet” may form the exterior of the body, and may be coupled to the outer side of the inner cabinet such that the heat insulating material is arranged between the inner cabinet and the outer cabinet.

[0065] “The heat insulating material” may insulate the inside of the storage chamber and the outside of the storage chamber such that the temperature inside the storage chamber can be maintained at a set appropriate temperature without being influenced by the outer environment of the storage chamber. According to an embodiment, the heat insulating material may include a foam heat insulating material. The foam heat insulating material may be molded by injecting and foaming urethane foam wherein polyurethane and a foaming agent are mixed between the inner cabinet and the outer cabinet.

[0066] According to an embodiment, the heat insulating material may additionally include a vacuum heat insulating material other than the foam heat insulating material, or the heat insulating material may consist only of a vacuum heat insulating material instead of the foam heat insulating material. The vacuum heat insulating material may include a core material, and an outer covering that houses the core material and seals the inside by vacuum or pressure close to vacuum. However, the heat insulating material is not limited to a foam heat insulating material or a vacuum heat insulating material as described above, and may include various materials that can be used for insulation.

[0067] “The storage chamber” may include a space limited by the inner cabinet. The storage chamber may further include an inner cabinet that limits the space corresponding to the storage chamber. In the storage chamber, various goods such as food, medicine, cosmetics, etc. may be stored, and the storage chamber may be formed such that at least one side is opened for putting in or taking out goods.

[0068] The refrigerator may include one or more storage chambers. When two or more storage chambers are formed in the refrigerator, each storage chamber may have different uses, and may be maintained at different temperatures. For this, each storage chamber may be partitioned from each other by a partition including a heat insulating material.

[0069] The storage chamber may be provided to be maintained in an appropriate temperature range according to uses, and may include “a refrigeration chamber,”“a freezer,” or “a temperature conversion chamber” that are divided according to the uses and / or the temperature ranges. The refrigeration chamber may be maintained at an appropriate temperature for keeping goods refrigerated, and the freezer may be maintained at an appropriate temperature for keeping goods frozen. “Refrigeration” may mean cooling goods to be cold within a limit of not freezing the goods, and as an example, the refrigeration chamber may be maintained in a range of 0° C. to 7° C. “Freezing” may mean cooling goods to be frozen or to be maintained in a frozen state, and as an example, the freezer may be maintained in a range of −20° C. to −1° C. The temperature conversion chamber may be used as any one of a refrigeration chamber or a freezer by the user's selection or regardless of it.

[0070] The storage chamber may be referred to by various names such as “a vegetable chamber,”“a fresh chamber,”“a cooling chamber,” and “an ice making chamber” other than the names such as “a refrigeration chamber,”“a freezer,” and “a temperature conversion chamber,” etc., and the terms such as “a refrigeration chamber,”“a freezer,” and “a temperature conversion chamber,” etc. used below should be understood as meaning that comprehensively includes storage chambers having uses and temperature ranges corresponding to each of them.

[0071] According to an embodiment, the refrigerator may include at least one door that is constituted to open and close the opened one side of the storage chamber. The door may be provided to open and close each of the one or more storage chambers, or may be provided such that one door opens and closes the plurality of storage chambers. The door may be rotatably or slidably installed on the front surface of the body.

[0072] “The door” may be constituted to seal the storage chamber when the door is closed. The door may include a heat insulating material like the body in order to insulate the storage chamber when the door is closed.

[0073] According to an embodiment, the door may include a door outer plate forming the front surface of the door, a door inner plate forming the rear surface of the door and facing the storage chamber, an upper cap, a lower cap, and a door heat insulating material provided in their insides.

[0074] On the rim of the door inner plate, a gasket that seals the storage chamber by being adhered to the front surface of the body when the door is closed may be provided. The door inner plate may include a dyke that projects toward the rear side such that a door bin that can keep goods is installed. Here, the door bin may be referred to as a door basket or a door bin.

[0075] According to an embodiment, the door may include a door body, and a front panel that is separably coupled to the front side of the door body and forms the front surface of the door. The door body may include a door outer plate forming the front surface of the door body, a door inner plate forming the rear surface of the door body and facing the storage chamber, an upper cap, a lower cap, and a door heat insulating material provided in their insides.

[0076] The refrigerator may be divided into a French door type, a side-by-side type, a bottom mounted freezer (BMF), a top mounted freezer (TMF), or a 1 door refrigerator, etc. according to the arrangement of the doors and the storage chambers.

[0077] According to an embodiment, the refrigerator may include a cold air supplying device that is provided to supply cold air to the storage chamber.

[0078] “The cold air supplying device” may include a machine, a tool, an electronic apparatus, and / or a system combining them that can cool the storage chamber by generating cold air and guiding the cold air.

[0079] According to an embodiment, the cold air supplying device may generate cold air through a freezing cycle including compression, condensation, expansion, and evaporation processes of a refrigerant. For this, the cold air supplying device may include a freezing cycle device including a compressor, a condenser, an expansion device, and an evaporator that can drive a freezing cycle. Also, according to an embodiment, the cold air supplying device may include a semiconductor such as a thermoelectric element. The thermoelectric element may cool the storage chamber with heat generating and cooling operations through a Peltier effect.

[0080] According to an embodiment, the refrigerator may include a machine chamber that is provided such that at least some components belonging to the cold air supplying device are arranged.

[0081] “The machine chamber” may be provided to be partitioned and insulated from the storage chamber for preventing transmission of heat generated from the components arranged in the machine chamber to the storage chamber. In order to radiate heat to the components arranged inside the machine chamber, the inside of the machine chamber may be constituted to communicate with the outside of the body.

[0082] According to an embodiment, the refrigerator may include a dispenser that is provided on the door to provide water and / or ice. The dispenser may be provided on the door such that a user can approach it without opening the door.

[0083] According to an embodiment, the refrigerator may include an ice making device that is provided to generate ice. The ice making device may include an ice making tray storing water, an ice moving device separating ice from the ice making tray, and an ice bucket storing ice generated in the ice making tray.

[0084] According to an embodiment, the refrigerator may include a controller for controlling the refrigerator.

[0085] “The controller” may include memory that stores or memorizes programs and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling the cold air supplying device, etc. according to the programs and / or the data memorized in the memory.

[0086] The memory stores or records various kinds of information, data, instructions, programs, etc. necessary for the operations of the refrigerator. The memory may memorize temporary data generated while a control signal for controlling the components included in the refrigerator is generated. The memory may include at least one of volatile memory or non-volatile memory or a combination of them.

[0087] The processor controls the overall operations of the refrigerator. The processor may control the components of the refrigerator by executing the programs stored in the memory. The processor may include a separate NPU that performs operations of an artificial intelligence model. Also, the processor may include a central processing unit, a graphic-dedicated processor (GPU), etc. The processor may generate a control signal for controlling the operations of the cold air supplying device. For example, the processor may receive temperature information of the storage chamber from the temperature sensor, and generate a cooling control signal for controlling the operations of the cold air supplying device based on the temperature information of the storage chamber.

[0088] Also, the processor may process a user input of a user interface according to the programs and / or the data memorized / stored in the memory, and control the operation of the user interface. The user interface may be provided by using an input interface and an output interface. The processor may receive a user input from the user interface. Also, the processor may transmit a display control signal for displaying an image on the user interface in response to a user input and image data to the user interface.

[0089] The processor and the memory may be provided integrally, or provided separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub processor. The memory may include one or more memories.

[0090] According to an embodiment, the refrigerator may include a processor and memory controlling all of the components included in the refrigerator, and include a plurality of processors and a plurality of memories individually controlling the components of the refrigerator. For example, the refrigerator may include a processor and memory controlling the operation of the cold air supplying device according to an output of the temperature sensor. Also, the refrigerator may separately include a processor and memory controlling the operation of the user interface according to a user input.

[0091] A communication module may communicate with external devices such as a server, a mobile device, another home appliance, etc. through an ambient access point (AP). The access point (AP) may connect a local area network (LAN) to which the refrigerator or a user device is connected to a wide area network (WAN) to which a server is connected. The refrigerator or the user device may be connected to the server through the wide area network (WAN).

[0092] An input interface may include a key, a touch screen, a microphone, etc. The input interface may receive a user input, and transmit it to the processor.

[0093] An output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated in the processor.

[0094] Meanwhile, in the disclosure, the meaning of the feature that the refrigerator 100“provides” information may not only include the feature of displaying information through a display 123 included in the refrigerator 100, but also the feature of transmitting information to a user terminal communicatively connected with the refrigerator 100 and displaying information through a display of the user terminal.

[0095] Hereinafter, the refrigerator according to various embodiments will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the disclosure. As illustrated in FIG. 1, the refrigerator 100 may include a camera 110, an outputter 120, a communication interface 130, a microphone 140, a sensor 150, memory 160, and a processor 170. The refrigerator 100 may be a device that is for keeping food or medicine, etc. at a predetermined temperature such that they are kept cool or do not rot. The refrigerator 100 according to an embodiment of the disclosure was illustrated in a form of a general household refrigerator, but is not limited thereto, and it may be a kimchi refrigerator, a liquor refrigerator, a cosmetic refrigerator, a freezer, etc.

[0096] The camera 110 is a component for generating a photographed image by photographing a subject, and here, a photographed image may include both of a moving image and a still image. Meanwhile, “an image” in the disclosure may be a concept including both of an image output on the display 123 and an image frame photographed by the camera 110. Also, in a photographed image, at least one object may be included. “An object” according to an embodiment of the disclosure is a component included in an image photographed through the camera, and may include a food object, a hand object, a face object, a kitchenware object, etc.

[0097] In particular, the camera 110 may photograph the storage chamber inside the body 230 of the refrigerator 100, and the door bin (or the door basket, the pantry) areas of the doors 210, 220. The camera 110 may be provided in at least one of the upper area, the lower area, or the side surface area inside the body 230 for photographing the inside of the body 230 and the door bin areas of the doors 210, 220. Also, the camera 110 may be provided outside the refrigerator 100 and photograph the outside of the refrigerator 100. In other words, the camera 110 can obviously be implemented not only as one camera, but also as a plurality of cameras depending on embodiments. More detailed explanation in this regard will be described with reference to FIG. 2A to FIG. 2E.

[0098] FIG. 2A is a diagram illustrating a refrigerator when the doors are opened according to an embodiment of the disclosure. The refrigerator 100 in FIG. 2A was illustrated such that a plurality of doors 210, 220 are provided on both sides of the upper part, but is not limited thereto, and the refrigerator 100 may be implemented as a French door type, a side-by-side type, etc. according to the arrangement of the doors 210, 220, the storage chamber 230, etc.

[0099] The refrigerator 100 may include doors 210, 220 on both sides of the upper part. Also, the refrigerator 100 may include a display 123 on one or more of the doors 210, 220 on both sides of the upper part. Here, on the display 123, not only information on the food stored in the refrigerator 100 but also various kinds of information (e.g., event information, notification information, recipe information, etc. received from the outside) may be provided.

[0100] The refrigerator 100 may further include a storage chamber 230 as illustrated in FIG. 2A. The storage chamber 230 may be opened by the doors 210, 220 that are opened and closed, and accommodate water, beverages, and refrigerated or frozen food. Here, the storage chamber 230 may include a plurality of accommodating spaces and storage spaces. The storage chamber 230 may be partitioned by partitions arranged inside the body. The storage chamber 230 may be divided into a freezing chamber arranged in the lower part of the refrigerator 100 and a refrigeration chamber arranged in the upper part. However, the arrangement of the freezing chamber and the refrigeration chamber is not limited thereto, and they may be arranged while their locations are changed with each other.

[0101] The doors 210, 220 may rotate by an angle set by a hinge (e.g., smaller than or equal to 300°), and open or close a portion of the front surface of the storage chamber 230.

[0102] Here, the second door 220 among the plurality of doors 210, 220 may include a display 123 that displays functions and the setting of the refrigerator 100 on the surface, and can be changed by a user's input (e.g., a touch or selection of a button). Other than the above, at least some doors among the plurality of doors 210, 220 may further include a dispenser that provides water, ice, or sparkling water and / or a handle that can be gripped, etc.

[0103] Meanwhile, as illustrated in FIG. 2A, a camera 110 may be included in the upper area of the body for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220.

[0104] FIG. 2B is an upper perspective view of the refrigerator 100, and as illustrated in FIG. 2B, the camera 110 may be arranged in the upper central area of the body (in particular, the top table) for photographing both of the door bins of the first door 210 and the second door 220. FIG. 2C is a cross-sectional view that cut the drawing in FIG. 2B by A-A′, and as illustrated in FIG. 2C, the camera 110 may be arranged to be toward a lower direction by a predetermined angle (e.g., 30 degrees) for photographing at least a part of the storage chamber 230 and the door bins of the doors 210, 220.

[0105] FIG. 2D and FIG. 2E are upper perspective views of the refrigerator 100, and are diagrams illustrating photographing ranges 240-1, 240-2 photographed by the camera 110. FIG. 2D is a diagram illustrating the photographing range 240-1 photographed by the camera 110 while the doors 210, 220 of the refrigerator 100 are fully opened, and FIG. 2E is a diagram illustrating the photographing range 240-2 photographed by the camera 110 while the doors 210, 220 of the refrigerator 100 satisfied a predetermined condition. Here “the predetermined condition” is a condition for obtaining an optimal door bin image, and here, “an optimal door bin image” may be an image that satisfies an optimal condition for identifying information on the food included in the door bins.

[0106] Also, the camera 110 may provide a photographed image to the processor 170 for managing putting in and taking out of food.

[0107] In addition, the camera 110 may be implemented as a wide angle camera for photographing a wide field of view, but is not limited thereto.

[0108] Meanwhile, in FIG. 2A to FIG. 2E, it was explained that the camera 110 is located in the upper part inside the body, but this is merely an example, and it is obvious that a plurality of cameras can be provided in other areas (e.g., a rear surface area, a lower area, a side surface area, etc.) inside the body.

[0109] The outputter 120 may provide various feedbacks. In particular, as illustrated in FIG. 1, the outputter 120 may include a speaker 121, light emitting diodes (LED) 122, a display 123, etc., but this is merely an example, and the outputter 120 may further include other outputters (e.g., a haptic provision device, etc.).

[0110] Here, the speaker 121 may be provided inside or outside the refrigerator 100, and provide various auditory feedbacks through audio. The LED 122 may be provided inside the storage chamber 230 or the doors 210, 220 inside the refrigerator 100, and provide various visual feedbacks through an indicator in a specific form (e.g., an arrow, etc.) and flickering, etc. The display 123 may be located in at least some areas of the plurality of doors 210, 220, and provide various visual feedbacks to the user.

[0111] In particular, the outputter 120 may output information guiding that food was put in(or stored into) or taken out(or removed from) of the refrigerator 100, information on the food that was put in or taken out, a food list, a candidate list, etc. Detailed explanation in this regard will be described with reference to the drawings later.

[0112] The communication interface 130 may perform communication with an external server or an external terminal device. In particular, the communication interface 130 may transmit an image including food to an external server for obtaining information on the food, and receive information on the food from the external server. Also, the communication interface 130 may transmit information on food and information on a storage location of food, etc. to a user terminal, and receive a control command from the user terminal. Here, the communication interface 130 may directly perform communication with the user terminal, but this is merely an example, and the communication interface 130 can obviously perform communication with an external user terminal through a server.

[0113] In particular, the communication interface 130 may perform communication with various types of external devices by using various wireless communication technologies or mobile communication technologies. As such wireless communication technologies, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB) communication, Zigbee, infrared Data Association (IrDA), or near field communication (NFC), etc. may be included, and as mobile communication technologies, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, etc. may be included.

[0114] The microphone 140 is a component that obtains an audio signal and converts it into an electric signal, and may be provided inside or outside the refrigerator 100. In particular, the microphone 140 may receive an audio signal including a user voice. Here, the user voice may include information on putting in or taking out (referred to as “putting in / taking out” hereinafter) and information on food (e.g., types of food, expiry dates of food, etc.).

[0115] The sensor 150 may detect an operation state (e.g.: power or a temperature) of the refrigerator 100, or an external environmental state (e.g.: a user state), and generate an electric signal or a data value corresponding to the detected state. In particular, the processor 170 may measure each of the temperatures of the plurality of storage chambers of the refrigerator 100 through sensing values obtained through the sensor 150. Alternatively, the processor 170 may recognize that the user approached through the sensor 150, and may control the camera 110 to be in a prepared state. Alternatively, the processor 170 may detect opening of the door through the sensor 150, and drive the camera 110.

[0116] The memory 160 may store an operating system (OS) for controlling overall operations of the components of the refrigerator 100, and instructions or data related to the components of the refrigerator 100. In particular, the memory 160 may store various components for managing putting in / taking out of food. Also, the memory 160 may store a food database (DB) that stores information on the food stored in the refrigerator 100 (e.g., types of food, the capacity(or net contents) of food, expiry dates of food, storage locations of food, etc.).

[0117] Also, according to an embodiment, the memory 160 may store a trained first neural network model (e.g., an object classification model, etc.) for obtaining feature information corresponding to food that was put in or taken out of the refrigerator 100. Alternatively, the memory 160 may store a trained second neural network model (e.g., an object recognition model, etc.) for recognizing food that was put in or taken out of the refrigerator 100.

[0118] Meanwhile, the memory 160 may be implemented as non-volatile memory (ex: a hard disc, a solid state drive (SSD), flash memory), volatile memory (the memory inside the processor 170 may be included), etc.

[0119] The processor 170 may control the refrigerator 100 according to the at least one instruction stored in the memory 160.

[0120] In particular, the processor 170 may include one or more processors. Specifically, the one or more processors may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or a random combination of other components of the refrigerator, and perform operations regarding communication or data processing. The one or more processors may execute one or more programs or instructions stored in the memory. For example, the one or more processors may perform the method according to an embodiment of the disclosure by executing the one or more instructions stored in the memory.

[0121] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor, or performed by a plurality of processors. In other words, when a first operation, a second operation, and a third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first processor, or the first operation and the second operation may be performed by the first processor (e.g., a generic-purpose processor), and the third operation may be performed by a second processor (e.g., an artificial intelligence-dedicated processor). For example, an operation for managing putting in / taking out of food may be performed through a generic-purpose processor such as a CPU, etc., and an operation of obtaining feature information corresponding to a food object or an operation for recognizing food by using a neural network model may be performed by an artificial intelligence-dedicated processor such as an NPU, etc.

[0122] The one or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including a plurality of cores (e.g., multicores of the same kind or multicores of different kinds). In case the one or more processors are implemented as multicore processors, each of the plurality of cores included in the multicore processors may include internal memory of the processor such as cache memory, on-chip memory, etc., and a common cache shared by the plurality of cores may be included in the multicore processors. Also, each of the plurality of cores (or some of the plurality of cores) included in the multicore processors may independently read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction, or the plurality of entire cores (or some of the cores) may be linked with one another, and read a program instruction for implementing the method according to an embodiment of the disclosure and perform the instruction.

[0123] In case the method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in the multicore processors, or they may be implemented by the plurality of cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to an embodiment, all of the first operation, the second operation, and the third operation may be performed by a first core included in the multicore processors, or the first operation and the second operation may be performed by the first core included in the multicore processors, and the third operation may be performed by a second core included in the multicore processors.

[0124] In the embodiments of the disclosure, the processor may mean a system on chip (SoC) wherein one or more processors and other electronic components are integrated, a single core processor, a multicore processor, or a core included in the single core processor or the multicore processor. Also, here, the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, etc., but the embodiments of the disclosure are not limited thereto.

[0125] According to an embodiment of the disclosure, if a trigger signal is detected, the processor 170 obtains an image by photographing at least a portion of the inside of the body and the doors 210, 220 through the camera 110. Here, the trigger signal is a signal for initiating a photographing operation of the camera 110, and may be generated by a user approach, opening of the doors, a user voice, etc. The processor 170 detects a first food object included in the obtained image. The processor 170 tracks the first food object and identifies whether the first food object is put in or taken out of the refrigerator. Here, “putting in” may be an operation of keeping food in the refrigerator, and “taking out” may be an operation of taking out food kept in the refrigerator. If it is identified that the first food object is put in, the processor 170 obtains first feature information corresponding to the first food object by inputting the image into the trained first neural network model. Here, the first neural network model may be an object classification model for obtaining feature information corresponding to food objects. The processor 170 matches the image and the obtained first feature information, and stores them in the food database.

[0126] According to an embodiment, the processor 170 may obtain information on the first food object by inputting the image into the trained second neural network model. Here, the second neural network model may be an object recognition model for recognizing food objects. The processor 170 may store the information on the food in the food database together with the image and the obtained first feature information. Here, the information on the first food object may include at least one of a type of the first food object, a product name of the first food object, the manufacturer of the first food object, or the capacity(or net contents) of the first food object.

[0127] According to an embodiment, the processor 170 may crop an area including the first food object from the image. Then, the processor 170 may obtain the first feature information corresponding to the first food object by inputting an area wherein the first food object is included into the trained first neural network model.

[0128] According to an embodiment, if it is identified that the first food object is taken out, the processor 170 may obtain second feature information corresponding to the first food object by inputting the image into the trained first neural network model. The processor 170 may identify whether food matched with the second feature information exists among the food stored in the food database. If it is identified that food matched with the second feature information exists, the processor 170 may delete the food matched with the second feature information from the food database.

[0129] According to an embodiment, the processor 170 may identify whether a second food object was put in within a predetermined time after it was identified that the first food object was taken out. If it is identified that the second food object was put in within the predetermined time, the processor 170 may identify whether the first food object and the second food object are matched. If it is identified that the first food object and the second food object are matched, the processor 170 may identify an event wherein the first food object is put in again, and store information related to the event wherein the first food object is put in again in the food database.

[0130] According to an embodiment, if it is identified that the second food object was not put in within the predetermined time or the first food object and the second food object are not matched, the processor 170 may identify whether food corresponding to the first food object exists among the food stored in the food database. If it is identified that food corresponding to the first food object exists, the processor 170 may delete the food corresponding to the first food object from the food database.

[0131] According to an embodiment, if it is identified that food corresponding to the first food object does not exist, the processor 170 may provide a candidate list for the first food object. Here, the candidate list may include at least one of n food objects corresponding to the most similar feature information to third feature information corresponding to the first food object, unspecified packaged objects, or food objects that were recently put in.

[0132] According to an embodiment, if it is identified that food corresponding to the first food object does not exist, the processor 170 may identify whether an unspecified packaged object that was put in before the predetermined time exists. If it is identified that the unspecified packaged object exists, the processor 170 may update the food database by determining that the first food object and the unspecified packaged object correspond.

[0133] According to an embodiment, if it is identified that the first food object is put in, the processor 170 may identify whether it is possible to obtain the first feature information corresponding to the first food object. If it is identified that it is impossible to obtain the first feature information corresponding to the first food object, the processor 170 may obtain information on an area corresponding to the first food object from the image. The processor 170 may match the image and the information on the area corresponding to the first food object, and store them in the food database.

[0134] According to an embodiment, the processor 170 may detect a hand object from the image. The processor 170 may obtain identification information on the detected hand object based on a plurality of pre-stored hand objects. The processor 170 may match the identification information on the hand object, the image, and the first feature information, and store them in the food database.

[0135] FIG. 3 is a flow chart for illustrating a method of managing putting in and taking out of food according to an embodiment of the disclosure.

[0136] The refrigerator 100 may detect a trigger signal in the operation S305. Here, the trigger signal is a signal for driving the camera 110, and may be, for example, a trigger signal generated after opening of the door is detected by a door opening / closing sensor, a trigger signal generated after a user approach is detected by a proximity sensor, and a trigger signal generated after a user input for driving the camera is input through the microphone 140. However, this is merely an example, and a trigger signal can obviously be generated by other methods.

[0137] The refrigerator 100 may obtain an image by driving the camera 110 in the operation S310. Driving the camera 110 is an operation of initiating an operation of the camera 110 so as to obtain an image, and may be expressed as activating the camera 110, or initiating the camera 110. Here, the refrigerator 100 may obtain an image based on initial set values of the camera (an exposure time, frames per second (FPS), a gain, etc.). Also, the refrigerator 100 may set a region of interest (ROI) area in the image. Here, the ROI area may be an area wherein an appearance that the user's hand or food is put in or taken out is identified.

[0138] Also, in case the camera 110 is located inside, regular illumination and a regular environment can be maintained according to the illumination inside the refrigerator 100, but in case the camera 110 is located outside, the photographing environment may change according to the outside environment (e.g., outside illumination, etc.). Accordingly, the refrigerator 100 may dynamically change the set values of the camera 110 based on the outside environment (e.g., the illumination direction, the illumination, etc.).

[0139] The refrigerator 100 may detect a food object in the operation S315. Here, the detected food object may be a moving food object (e.g., a food object gripped by the user's hand, a food object put in dishware such as a pot, a food object covered by another food object, etc.). Specifically, the refrigerator 100 may recognize a user operation by using a trained neural network model. Here, the trained neural network model is an operation recognition model trained to recognize hand operations of the user, and hand operations of the user can be learned by images photographed by various outside environments. The refrigerator 100 may detect a moving food object and a hand object by recognizing hand operations of the user.

[0140] According to an embodiment, the refrigerator 100 may crop an area including the detected food object from the photographed image. The refrigerator 100 may obtain the first feature information corresponding to the first food object by inputting the area wherein the first food object is included into the trained first neural network model. Alternatively, the refrigerator 100 may obtain information on the first food object by inputting the area wherein the first food object is included into the trained second neural network model.

[0141] Also, the refrigerator 100 may detect a pantry object, a face object, a kitchenware object, etc. other than a hand object and a food object.

[0142] The refrigerator 100 may track the food objects in the operation S320. Specifically, the refrigerator 100 may detect the food objects for each of the image frames obtained by the camera 110 and track the objects. Here, the refrigerator 100 may analyze the moving directions of the food objects based on the locations of the food objects included in the plurality of image frames.

[0143] According to an embodiment, as illustrated in FIG. 4, the refrigerator 100 may detect the moving directions of the food object 410 and the hand object 420 included in the plurality of image frames 400-1 to 400-6. Here, the refrigerator 100 may detect a center point of the food object 410 included in the image (in particular, the cropped area), and detect the moving direction of the food object 410 according to moving of the center point. Then, the refrigerator 100 may identify in which area of the refrigerator 100 the food object 410 was stored through the moving direction of the food object 410 detected in the first to fourth image frames 400-1 to 400-4. Then, as the food object 410 detected in the first to fourth image frames 400-1 to 400-4 was not detected in the fifth and sixth image frames 400-5, 400-6, the refrigerator 100 may determine an event wherein food is put in.

[0144] Meanwhile, in the aforementioned embodiment, it was explained that the food object 410 is detected and the food object is tracked, but this is merely an example, and the refrigerator 100 may track a pantry object, a face object, a kitchenware object, etc. other than a hand object and a food object.

[0145] The refrigerator 100 may determine whether a food object is put in or taken out in the operation S325. According to an embodiment, the refrigerator 100 may analyze a food object included in the plurality of image frames, and determine whether the food object is put in or taken out.

[0146] According to an embodiment, if an event wherein a food object was included in some image frames among the plurality of images frames, and then the food object is not included in the remaining image frames is detected, the refrigerator 100 may determine that the food object was put in. Alternatively, if an event wherein a food object was not included in some image frames among the plurality of images frames, and then the food object is included in the remaining image frames is detected, the refrigerator 100 may determine that the food object was taken out.

[0147] According to an embodiment, the refrigerator 100 may determine whether a food object was put in or taken out based on a moving direction of the food object included in the plurality of image frames. In other words, if it is identified that a food object included in the plurality of image frames moves to the direction of the inside of the body or the door bin, the refrigerator 100 may determine that the food object was put in. Alternatively, if it is identified that a food object included in the plurality of image frames moves to the direction of the outside of the refrigerator 100, the refrigerator 100 may determine that the food object was taken out.

[0148] Meanwhile, while detecting a food object and tracking it, the refrigerator 100 may obtain information on the food object by inputting a photographed image (in particular, an area corresponding to the cropped food object) into the trained second neural network model (an object recognition model). Here, the information on the food object may include at least one of the type of the food object, the product name of the food object, the manufacturer of the first food object, or the capacity of the first food object, but is not limited thereto.

[0149] If putting in of a food object is determined, the refrigerator 100 may input an image into the trained first neural network model in the operation S330. Here, the first neural network model may be a neural network model trained to obtain feature information corresponding to a food object by inputting an image. Here, the first neural network model may be trained by a contrastive learning method. The contrastive learning method is a main learning method of self-supervised learning, and is a method of learning such that feature information (or feature values) corresponding to similar images is located to be close to each other in a vector space, and feature information corresponding to different images is located to be far away from each other. By classifying objects through a neural network model trained by the contrastive learning method as above, even if parts of food objects are covered by a hand object, or food objects are photographed by different angles, or parts of food objects are deformed, or food objects are packaged by different packaging methods, correct classification of the objects can become possible by the first neural network model.

[0150] According to an embodiment, the first neural network model may subdivide a label unit and apply it to learning, and here, a label input by the user may be applied to learning of the first neural network model.

[0151] The refrigerator 100 may obtain first feature information from the first neural network model in the operation S335. Here, the first feature information may include a feature value corresponding to a food object. Here, the first feature information may be used for identifying food that will be taken out later.

[0152] The refrigerator 100 may match an image and the first feature information in the operation S340. In other words, the refrigerator 100 may match an image (in particular, an area corresponding to a food object) and the first feature information, and store them. Not only that, the refrigerator 100 may match information on a food object obtained by the second neural network model with an image and the first feature information, and store them. As an example, the refrigerator 100 may store information on a food object such as “the name of the food, the storage space of the food, the date of putting in of the food, the data of taking out of the food, the type of the food, barcode information of the food, a text included in the food,” etc. together with the image and the first feature information.

[0153] The refrigerator 100 may update the food DB in the operation S360. Specifically, the refrigerator 100 may match an image, the first feature information, and information on food with the food DB, and store them in the food DB. Here, the refrigerator 100 may store the first feature information in a feature information list of the food DB. In particular, the refrigerator 100 may update the feature information list by comparing similarity of the first feature information newly stored in the feature information list and other feature information, and aligning the first feature information through the comparison result. Not only that, the feature information list may align the feature information by considering user information such as a user pattern, time of putting in / taking out, etc. as well as the feature information.

[0154] If taking out of a food object is determined, the refrigerator 100 may input the image into the trained first neural network model in the operation S345. Here, as the first neural network model is identical to the first neural network model explained earlier, overlapping explanation will be omitted.

[0155] The refrigerator 100 may obtain second feature information from the first neural network model in the operation S350. Here, the second feature information may include a feature value corresponding to a food object. Also, although the second feature information was described as different information from the first feature information for distinguishing putting in / taking out, it may be the same value as the first feature information (or, a value within a range wherein it is determined as the same food type).

[0156] The refrigerator 100 may identify food matched with the second feature information in the operation S355. Specifically, the refrigerator 100 may identify food corresponding to the closest feature information to the second feature information among the feature information stored in the food DB stored in the refrigerator 100. Also, the refrigerator 100 may identify the food corresponding to the closest feature information to the second feature information as the food matched with the second feature information. In other words, as the feature information indicates a vector value within a vector space, the refrigerator 100 may calculate a distance between the second feature information and another feature information, and obtain information on the similarity of the second feature information and the another feature information.

[0157] Meanwhile, the refrigerator 100 may not only obtain food matched with the closest feature information to the second feature information, but also obtain N pieces of candidate feature information based on similarity between the second feature information and another feature information, and obtain a candidate list including the N pieces of candidate feature information. Here, N may be set in advance, but this is merely an example, and it may be changed by the user setting. The refrigerator 100 may receive input of a user command selecting food taken out to the user by using the candidate list.

[0158] The refrigerator 100 may update the food DB in the operation S360. Specifically, the refrigerator 100 may delete the food matched with the second feature information among the food stored in the food DB. Alternatively, the refrigerator 100 may change the information on the food matched with the second feature information among the food stored in the food DB. For example, the refrigerator 100 may change the number of the food stored in the food DB, etc.

[0159] The refrigerator 100 may provide a food list in the operation S365. Specifically, the refrigerator 100 may provide a food list based on the updated food DB. Here, the food list may include information on the food put into the plurality of areas of the refrigerator 100. In particular, the food list may include information on the food that was divided into each of the plurality of areas of the refrigerator 100 and stored in each of the plurality of areas. For example, the food list may divide information on the food stored in the left door bin, information on the food stored in the right door bin, information on the food stored in the first floor area inside the body, information on the food stored in the second floor area inside the body, and information on the food stored in the third floor area inside the body, and provide them to the user. Also, the food list may include information on the food that was put in, or information on the food that was taken out. Here, the information on the food that was put in or taken out may include information on the time of putting in or taking out, or the number of times of putting in or taking out, etc.

[0160] Meanwhile, according to the disclosure, the food DB may include a plurality of types of DBs. As an example, a first food DB may be a food DB including information on the food currently stored in the refrigerator 100. In other words, when the first food is put in, information on the first food may be added to the first food DB, and when the second food is taken out, information on the second food may be deleted from the first food DB. The first food DB may be used for managing the food currently stored in the refrigerator 100, and may also be used for providing a food list. As another example, a second food DB may be a food DB that temporarily stores information on the food currently taken out from the refrigerator 100 during a predetermined time. Here, the second food DB may be a food DB used for determining whether food taken out from the refrigerator 100 is put in again. As still another example, a third food DB may be a food DB storing information on the history of putting in / taking out of food. In other words, the third food DB may accumulate information on the history that food is put in, the history that food is taken out, etc. for each time zone, and store them. The third food DB may store all the histories of putting in / taking out of food, and may be used for the user to identify the history of putting in / taking out of food. Meanwhile, the first food DB may be referred to as “a food DB,” the second food DB may be referred to as “a temporarily stored food DB,” and the third food DB may be referred to as “a food history DB.”

[0161] Also, the refrigerator 100 may provide the history of taking out to the user by using a food history DB other than the food list. As an example, as illustrated in FIG. 5, the refrigerator 100 may provide a list 510 including food that was processed to have been taken out. Here, the list 510 including the food that was processed to have been taken out may be aligned for each type of the food as illustrated in FIG. 5, but this is merely an example, and the list 510 can obviously be aligned for each processing time of taking out.

[0162] The refrigerator 100 may update the food DB by using the list 510 including the food that was processed to have been taken out. Specifically, if a user command for selecting one food in the list 510 including the food that was processed to have been taken out which was illustrated in FIG. 5 and correcting the processing of taking out is input, the refrigerator 100 may provide a UI 610 for correcting the processing of taking out for the selected food. On the UI 610 for correcting the processing of taking out, a candidate list including a plurality of pieces of candidate food and a put back item may be provided. If one piece of candidate food is selected in the food list by the user, the refrigerator 100 may correct the processing of taking out for the food selected in the list 510, and perform the processing of taking out for the selected candidate food on the UI 610. By this, the refrigerator 100 may perform update such that the food selected in the list 510 is added to the food DB, and perform update such that the candidate food selected on the UI 610 is deleted from the food DB. Alternatively, if the “put back” item on the UI 610 for correcting the processing of taking out is selected, the refrigerator 100 may perform update such that the food selected in the list 510 is added to the food DB.

[0163] According to an embodiment of the disclosure, FIG. 7 and FIG. 8 are diagrams illustrating a food history list. Here, the food history list may be generated based on the food history DB.

[0164] The first list 710 illustrated in FIG. 7 may be a list that aligned information on food that was put in or taken out of the refrigerator 100 per date. Here, in the first list 710, the history of putting in / taking out of the food that was put in or taken out, information on the food that was put in or taken out, and images of the food that was put in or taken out may be included, as illustrated in FIG. 7. Also, the first list 710 may be aligned (or classified) by various standards. For example, the first list 710 may be aligned based on the dates of putting in / taking out by using the first UI element 720 in FIG. 7, and may be aligned by the names of the food and the categories of the food. Also, the first list 710 may be corrected according to a user input. For example, the refrigerator 100 may delete at least one selected among the food included in the first list 710 based on a user command input through the second UI element 730 in FIG. 7. Here, the first and second UI elements 720, 730 may be displayed on the first list 710.

[0165] Also, the first list 710 may be updated to include only the food that was put in or the food that was taken out. For example, if a user command for providing the second list including only the food that was put in (e.g., a user command selecting the “IN” item 715 illustrated in FIG. 7) is input, the refrigerator 100 may provide the second list 810 including only the food that was put in among the food included in the first list 710, as illustrated in FIG. 8.

[0166] Hereinafter, various embodiments of the disclosure will be explained with reference to the drawings. Meanwhile, as a method of identifying putting in or taking out of food through a photographed image was explained in detail in FIG. 3, overlapping explanation will be omitted in the following embodiments.

[0167] FIG. 9 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure.

[0168] The refrigerator 100 may identify taking out of food in the operation S905. Specifically, the refrigerator 100 may determine taking out of food detected from an image based on an image photographed by the camera 110, and obtain feature information corresponding to the detected food. Here, the refrigerator 100 may temporarily store information on the food that was taken out in a temporarily stored food DB. When a predetermined time passes, the refrigerator 100 may delete the information on the food that was taken out which was temporarily stored in the temporarily stored food DB.

[0169] The refrigerator 100 may identify whether putting in of food was generated within the predetermined time in the operation S910. In other words, the refrigerator 100 may identify whether putting in of food was generated within the predetermined time based on the time when taking out of food was identified (or the time of taking out). As an example, the refrigerator 100 may identify whether putting in of food was generated within the predetermined time based on the time of taking out for each of the plurality of pieces of food that were taken out. Here, the predetermined time may be a reference time for determining whether food was put in again, and may be, for example, one hour. Here, the predetermined time may be set according to a user input.

[0170] If it is identified that putting in of food was generated within the predetermined time in the operation S910-Y, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists in the operation S915. Here, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists based on feature information corresponding to the food that was put in within the predetermined time and feature information corresponding to the food that was taken out which is stored in the temporarily stored food DB. In other words, the refrigerator 100 may obtain (or calculate) the similarity (or the distance) between the feature information corresponding to the food that was put in within the predetermined time and the feature information corresponding to the food that was taken out which is stored in the temporarily stored food DB. Then, the refrigerator 100 may identify whether food that was taken out which is matched with the food that was put in within the predetermined time exists based on the obtained similarity (or distance).

[0171] If it is identified that food that was taken out which is matched with the food that was put in within the predetermined time exists in the operation S915-Y, the refrigerator 100 may determine that food was put in again in the operation S920. Here, putting in of food again means that food that was previously put in the refrigerator 100 is put in again within the predetermined time after being taken out, and it may be distinguished from the initial putting in.

[0172] The refrigerator 100 may update the food DB according to putting in of food again in the operation S925. Specifically, if it is determined that food was put in again, the refrigerator 100 may delete the information on the food that was put in again which is stored in the temporarily stored food DB, and store again the information on the food that was put in again in the food DB. Also, the refrigerator 100 may maintain the time of putting in of the food that was put in again as the initial time of putting in but not the time of putting in again, and update the food DB such that the history of putting in again (the time of taking out and the time of putting in again, etc.) is added.

[0173] If it is identified that putting in of food is not generated within the predetermined time in the operation S910-N, or if it is identified that food that was taken out which is matched with the food that was put in within the predetermined time does not exist in the operation S915-N, the refrigerator 100 may compare the food that was taken out and the food DB in the operation S930. In other words, the refrigerator 100 may obtain (or calculate) the similarity (or the distance) by comparing the feature information corresponding to the food that was taken out and the feature information of the food stored in the food DB.

[0174] The refrigerator 100 may identify whether food matched with the food that was taken out exists in the operation S935. Specifically, the refrigerator 100 may identify whether food having feature information of which similarity to the feature information corresponding to the food that was taken out is greater than or equal to a threshold value (or the distance is smaller than or equal to a threshold value) exists.

[0175] If it is identified that food matched with the food that was taken out exists in the operation S935-Y, the refrigerator 100 may determine taking out of food in the operation S940. In other words, the refrigerator 100 may determine taking out of food that was taken out in the operation S905.

[0176] The refrigerator 100 may update the food DB according to taking out of food in the operation S940. In other words, the refrigerator 100 may delete or change the information on the food matched with the food that was taken out among the food stored in the food DB.

[0177] If it is identified that food matched with the food that was taken out does not exist in the operation S935-N, the refrigerator 100 may provide a candidate list in the operation S950. Here, the candidate list is a list provided to the user for identifying food that was taken out, and may include at least one of n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in.

[0178] The refrigerator 100 may perform processing of taking out according to a user input in the operation S955. Specifically, if a user input selecting one of the plurality of pieces of food included in the candidate list is received, the refrigerator 100 may perform processing of taking out for the selected food (or unspecified packaged food).

[0179] The refrigerator 100 may update the food DB according to the processing of taking out in the operation S960. In other words, the refrigerator 100 may delete or change the information on the food matched with the food selected by the user among the food stored in the food DB.

[0180] According to an embodiment, if putting in of food is generated, the refrigerator 100 may determine whether putting in of the food is putting in of the food again by aligning a take-out list of food within the predetermined time from the time point of generation of putting in. Here, the take-out list of food may be a list generated based on the temporarily stored food DB. For example, the predetermined time may be one hour, but this is merely an example, and it may be changed according to a user input. Alternatively, the predetermined time may be changed according to a use pattern of the user. For example, the refrigerator 100 may obtain pattern information for the time when the same food object is put in after a food object is taken out, and set (or change) the predetermined time according to the pattern information. For example, if it is determined that the time pattern that the user puts in a food object again after taking out the food object is 10 minutes, the refrigerator 100 may set the predetermined time as 10 minutes.

[0181] FIG. 10 is a diagram for illustrating an embodiment wherein food is put in again within a predetermined time according to an embodiment of the disclosure. Meanwhile, each of the first to third images explained in FIG. 10 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0182] The refrigerator 100 may identify that milk was taken out at 15:10 on Aug. 10, 2023 through a first image 1010.

[0183] Then, the refrigerator 100 may obtain a second image 1020. Here, the refrigerator 100 may identify whether milk that was taken out was put in again within the predetermined time. In other words, the refrigerator 100 may identify that milk was put in through the second image 1020, and identify that the time when the milk was put in was 15:43 on Aug. 10, 2023 which is within the predetermined time (e.g., 1 hour) from the time point when the milk was taken out. Accordingly, the refrigerator 100 may identify that the milk was put in again through the second image 1020 and the time of putting in. Here, the refrigerator 100 may update the food DB according to putting in of the milk again. Specifically, the refrigerator 100 may delete the information on the milk that was put in again which was stored in the temporarily stored food DB, and store again the information on the milk that was put in again in the food DB. Also, the refrigerator 100 may maintain the time of putting in of the milk that was put in again as the initial time of putting in but not the time of putting in again, and update the food DB such that the history of putting in again (the time of taking out and the time of putting in again, etc.) is added.

[0184] Also, the refrigerator 100 may obtain a third image 1030. Here, the refrigerator 100 may identify whether milk that was taken out was put in again within the predetermined time. Here, the refrigerator 100 may identify that milk was put in through the third image 1030, and identify that the time when the milk was put in was 17:13 on Aug. 10, 2023 which passed the predetermined time (e.g., 1 hour) from the time point when the milk was taken out. Accordingly, the refrigerator 100 may identify that new milk was initially put in but not that milk was put in again through the third image 1030 and the time of putting in. Here, the refrigerator 100 may update the food DB according to putting in of new milk. Specifically, the refrigerator 100 may store information on the new milk and information on the initial time of putting in in the food DB.

[0185] By a method as above, the refrigerator 100 may manage the food DB by identifying putting in of food or putting in of food again, and store the history regarding putting in or putting in again in the food history DB.

[0186] FIG. 11 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure.

[0187] The refrigerator 100 may identify taking out of food in the operation S1110. Specifically, the refrigerator 100 may determine taking out of food detected from an image based on an image photographed by the camera 110, and obtain feature information corresponding to the detected food.

[0188] The refrigerator 100 may identify whether food that is matched exists by comparing the food that was taken out and the food DB in the operation S1120. Specifically, the refrigerator 100 may identify whether food having feature information of which similarity to the feature information corresponding to the food that was taken out is greater than or equal to a threshold value (or the distance is smaller than or equal to a threshold value) exists among the food stored in the food DB.

[0189] If it is identified that food matched with the food that was taken out exists in the operation S1120-Y, the refrigerator 100 may determine taking out of food in the operation S1130. Here, the refrigerator 100 may update the food DB according to taking out of food. In other words, the refrigerator 100 may delete or change the information on the food matched with the food that was taken out among the food stored in the food DB.

[0190] If it is identified that food matched with the food that was taken out does not exist in the operation S1120-N, the refrigerator 100 may provide a candidate list in the operation S1150. Here, the candidate list is a list provided to the user for identifying food that was taken out, and may include at least one of n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in. Here, the unspecified packaged food is food that is put in in an unspecified packaged state, and may be, for example, food that is put in in an unspecified packaged state of which content cannot be identified such as a black plastic bag, a shopping bag, a box, etc.

[0191] The refrigerator 100 may receive a user input selecting unspecified packaged food in the operation S1160. In other words, the refrigerator 100 may receive a user input selecting (or touching) unspecified packaged food among a plurality of pieces of food included in a displayed candidate list.

[0192] The refrigerator 100 may match the unspecified packaged food and the food that was taken out in the operation S1170. Specifically, the refrigerator 100 may match the information on the food that was taken out and information on the unspecified packaged food.

[0193] The refrigerator 100 may output an inquiry message to the user in the operation S1180. Specifically, the refrigerator 100 may output an inquiry message that checks whether the food that was taken out is matched with the unspecified packaged food to the user. For example, the refrigerator 100 may output an inquiry message which is “Is it correct that the apples that were taken out had been put in in a black plastic bag at 3:30 on August 10, 2023?” Here, the inquiry message may be provided through the display 123, but this is merely an example, and it may be provided through the speaker 121 in an auditory form. The refrigerator 100 may receive a user input for a response to the inquiry message.

[0194] The refrigerator 100 may update the food DB in the operation S1190. Specifically, if it is identified that the food that was taken out and the unspecific packaged food are matched (i.e., if a positive feedback input of the user for the inquiry message is received), the refrigerator 100 may delete or change the information on the unspecific packaged food stored in the food DB, and update the information on the unspecific packaged food stored in the food history DB to the information on the food that was taken out. If it is identified that the food that was taken out and the unspecific packaged food are not matched (i.e., if a negative feedback input of the user for the inquiry message is received), the refrigerator 100 may output the candidate list for inquiring about the food that was taken out again.

[0195] FIG. 12 is a diagram for illustrating an embodiment of managing unspecified packaged objects according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 12 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0196] The refrigerator 100 may identify that unspecified packaged food (e.g., food in a black plastic bag) was put in at 17:13 on Aug. 10, 2023 through a first image 1210. The refrigerator 100 may store information on the unspecified packaged food in the food DB.

[0197] Then, the refrigerator 100 may obtain a second image 1220. Here, the refrigerator 100 may identify that food that is taken out is apples through the second image 1220, and identify that apples were taken out at 17:15 on Aug. 10, 2023. The refrigerator 100 may identify whether apples exist in the food DB. If it is identified that apples do not exist in the food DB, the refrigerator 100 may provide a candidate list. The refrigerator 100 may identify that the apples correspond to the unspecified packaged food based on a user input that was input through the candidate list. The refrigerator 100 may update the unspecified packaged food stored in the food DB to apples, and update the food DB so as to remove the apples or reduce the number of the apples which are the unspecified packaged food stored in the food DB. Here, the refrigerator 100 may provide an inquiry message to the user for identifying the number of the apples stored in the unspecified packaged food. Alternatively, the refrigerator 100 may count the number of the apples without deleting the information on the unspecified packaged food in the food DB. Then, if it is identified that the unspecified packaged food (e.g., a black plastic bag) is taken out, the refrigerator 100 may delete the information on the unspecified packaged food in the food DB, and store the counted number of the apples in the food history DB.

[0198] FIG. 13 is a flow chart for illustrating a method of managing taking out of food according to an embodiment of the disclosure. Meanwhile, as the operations S1310 to S1330 disclosed in FIG. 13 correspond to the operations S1110 to S1130 disclosed in FIG. 11, overlapping explanation will be omitted.

[0199] The refrigerator 100 may identify whether unspecified packaged food that was recognized to have been put in before a predetermined time exists in the operation S1340. Here, the predetermined time may be a week, but is not limited thereto.

[0200] If it is identified that unspecified packaged food that was recognized to have been put in before the predetermined time exists in the operation S1340-Y, the refrigerator 100 may match the unspecified packaged food and food that was taken out in the operation S1350. Specifically, the refrigerator 100 may match information on the food that was taken out and the information on the unspecified packaged food with each other.

[0201] The refrigerator 100 may update the food DB in the operation S1360. Specifically, the refrigerator 100 may update the information on the unspecified packaged food stored in the food DB and the food history DB to the information on the food that was taken out.

[0202] The refrigerator 100 may perform processing of taking out of the food in the operation S1370. In other words, the refrigerator 100 may delete or change the information on the unspecified packaged food stored in the food DB.

[0203] If it is identified that unspecified packaged food that was recognized to have been put in before the predetermined time does not exist in the operation S1340-N, the refrigerator 100 may update the food DB in the operation S1380. Specifically, the refrigerator 100 may update information on taking out of the food that was taken out in the food DB. Alternatively, the refrigerator 100 may provide a candidate list for identifying the type of the food that was taken out.

[0204] FIG. 14 is a flow chart for illustrating a method of managing putting in and taking out of bundled food according to an embodiment of the disclosure.

[0205] The refrigerator 100 may detect putting in of bundled food A in the operation S1410. Here, bundled food may be food wherein a plurality of pieces of single food are stored in a form of a bundle. Here, the second neural network model may be trained by designating a label for bundled food at the time of training, and the refrigerator 100 may identify whether food is bundled food by using the second neural network model. In other words, the refrigerator 100 may obtain, as information on the bundled food A, information on whether it is bundled food, and information on the type of the bundled food, the number of pieces of food included in the bundled food, etc. Also, the refrigerator 100 may obtain feature information of the bundled food by using the first neural network model.

[0206] The refrigerator 100 may update the food DB in the operation S1420. In other words, the refrigerator 100 may store the information on the bundled food A in the food DB. In particular, the refrigerator 100 may store the information on the number of the pieces of single food corresponding to the bundled food A and the feature information of the bundled food A in the food DB.

[0207] The refrigerator 100 may detect taking out of the single food a in the operation S1430. In other words, the refrigerator 100 may detect taking out of the single food a constituting the bundled food A. In particular, the refrigerator 100 may obtain, as information on the single food a, information on the type of the single food and the number of the pieces of the single food that were taken out, etc.

[0208] The refrigerator 100 may obtain feature information of the single food a in the operation S1440. Here, the refrigerator 100 may obtain the feature information of the single food a by using the first neural network model. The first neural network model may be trained such that the feature information of the bundled food A and the feature information of the single food a correspond.

[0209] Then, the refrigerator 100 may match the bundled food A and the single food a based on the obtained feature information in the operation S1450. Specifically, the refrigerator 100 may identify that the bundled food A and the single food a are of the same type of food based on the feature information of the bundled food A and the feature information of the single food a, and match the bundled food A and the single food a based on the identification result.

[0210] The refrigerator 100 may update the food DB in the operation S1460. Here, the refrigerator 100 may update the food DB based on the number of the pieces of the single food a included in the bundled food A and the number of the pieces of the single food a that were taken out. In case the number of the pieces of the single food a included in the bundled food A and the number of the pieces of the single food a that were taken out are identical, the refrigerator 100 may delete the information on the bundled food A stored in the food DB. In case the number of the pieces of the single food a that were taken out is smaller than the number of the pieces of the single food a included in the bundled food A, the refrigerator 100 may change the number of the pieces of the single food a included in the bundled food A stored in the food DB.

[0211] FIG. 15 is a diagram for illustrating an embodiment of managing putting in and taking out of bundled food according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 15 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0212] The refrigerator 100 may identify that the bundled food A was put in at 17:00 on Mar. 24, 2023 through a first image 1510. The refrigerator 100 may obtain information on the bundled food A through the second neural network model. Here, the refrigerator 100 may store, as information on the bundled food A, the type of the bundled food (i.e., eggs) and the number of the pieces of the bundled food (e.g., seven) in the food DB. Also, the refrigerator 100 may obtain feature information for the bundled food A through the first neural network model.

[0213] Then, the refrigerator 100 may obtain a second image 1520. Here, the refrigerator 100 may identify food a that is taken out through the second image 1520, and identify that eggs were taken out at 18:40 on Mar. 24, 2023. The refrigerator 100 may obtain information on the single food a through the second neural network model. Here, the refrigerator 100 may obtain, as information on the single food a, the type of the single food (i.e., eggs) and the number of the pieces of the single food that were taken out (e.g., one). Also, the refrigerator 100 may obtain feature information for the single food a through the first neural network model. The refrigerator 100 may identify that the bundled food A and the single food a are matched food based on the feature information for the bundled food A and the feature information for the single food a, and change the number of the pieces of the bundled food stored in the food DB based on the identification result. In other words, the refrigerator 100 may change the number of the pieces of the bundled food stored in the food DB from seven to six. Accordingly, the refrigerator 100 can provide more correct information on the food inside the refrigerator 100 by integrally managing the bundled food A and the single food a.

[0214] FIG. 16 to FIG. 17B are flow charts for illustrating a method of managing putting in of food according to whether specific information was obtained according to an embodiment of the disclosure. Meanwhile, each of the first and second images explained in FIG. 17B may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0215] The refrigerator 100 may determine putting in of food in the operation S1610. In other words, the refrigerator 100 may determine putting in of food based on a moving direction of a food object included in an image.

[0216] The refrigerator 100 may identify whether feature information for the food that was put in can be obtained in the operation S1620. Specifically, if an area of food gets to be covered by greater than or equal to a specific size by the user's hand while the food is being put in the refrigerator, the refrigerator 100 cannot obtain feature information corresponding to the food from an image. According to an embodiment, the refrigerator 100 may identify whether feature information for food that was put in can be obtained based on the size of the food included in an image. According to another embodiment, the refrigerator 100 may obtain feature information for food by inputting an image (or an area corresponding to the food) into the first neural network model. Then, the refrigerator 100 may identify whether food corresponding to the feature information exists (i.e., whether similar feature information exists). If it is identified that food corresponding to the feature information does not exist, the refrigerator 100 may identify that feature information for the food that was put in cannot be obtained.

[0217] If it is identified that feature information for the food that was put in can be obtained in the operation S1620-Y, the refrigerator 100 may match the image and the feature information for the food and store them in the food DB in the operation S1630, as explained above.

[0218] If it is identified that feature information for the food that was put in cannot be obtained in the operation S1620-N, the refrigerator 100 may obtain information on an area corresponding to the food object in the operation S1640. Specifically, the refrigerator 100 may obtain, as information on an area corresponding to the food object, texture information or information wherein the area corresponding to the food object was segmented from the image. Specifically, as illustrated in FIG. 17A, in case an image 1710 was obtained, the refrigerator 100 may segment an area 1720 corresponding to a food object, and obtain information on an area corresponding to the food object.

[0219] The refrigerator 100 may match the image and the information on the area corresponding to the food object, and store them in the operation S1650.

[0220] Afterwards, if food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may identify the food that was taken out based on an area corresponding to the food object.

[0221] Specifically, as illustrated in FIG. 17B, the refrigerator 100 may obtain a first image 1710, and identify that food was put in at 17:13 on Aug. 10, 2023 through the first image 1710. However, the refrigerator 100 may identify that feature information for the food cannot be obtained as the food was covered by the user's hand. The refrigerator 100 may obtain information on the area corresponding to the food, and store it.

[0222] Then, the refrigerator 100 may obtain a second image 1720. Here, the refrigerator 100 may identify that food that is taken out is milk through the second image 1720, and identify that milk was taken out at 17:15 on Aug. 10, 2023. According to an embodiment, the refrigerator 100 may identify that the milk that was taken out is the food for which feature information could not be obtained which was put in earlier, based on the information on the area corresponding to the food (texture information or segmented image information). The refrigerator 100 may update the food DB by matching the food for which feature information could not be obtained and the food that was taken out (milk). According to another embodiment, if milk which is food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may generate a candidate list, as explained above. Here, the candidate list may include food for which feature information could not be obtained other than n pieces of food corresponding to the most similar feature information to the feature information corresponding to the food that was taken out, unspecified packaged food, or food that was recently put in. If food for which feature information could not be obtained (i.e., an area corresponding to the food object) is selected through the candidate list, the refrigerator 100 may update the food DB by matching the food for which feature information could not be obtained and the food that was taken out. According to another embodiment, if milk which is food that does not have a history of having been put in in the food DB is taken out, the refrigerator 100 may ask a question to the user through the speaker 121. For example, the refrigerator 100 may output a question message which is “Is the food that was put in the refrigerator at 17:13 on Aug. 10, 2023 milk?”, and update the food DB by matching the food for which feature information could not be obtained and the food that was taken out through the user's feedback (e.g., “Yes”). Here, the refrigerator 100 may output the question message to the user based on the time of putting in of the food for which feature information could not be obtained and the time of taking out of the food that was taken out.

[0223] Also, the refrigerator 100 may train the first neural network model based on an image including an area corresponding to a food object and food that was taken out. In other words, the refrigerator 100 may train the first neural network model to obtain feature information corresponding to food that was taken out when an area corresponding to a food object is input into the first neural network model.

[0224] FIG. 18 is a flow chart for illustrating a method of managing putting in of food according to identification information corresponding to a hand object according to an embodiment of the disclosure.

[0225] Putting in of food may be generated in the operation S1810. Here, the refrigerator 100 may photograph an image regarding putting in of the food.

[0226] The refrigerator 100 may detect a hand object and a food object from the photographed image in the operation S1820.

[0227] The refrigerator 100 may separate an area corresponding to the detected hand object in the operation S1830. In other words, the refrigerator 100 may separate an area corresponding to the hand object from among the hand object and the food object detected from the photographed image.

[0228] The refrigerator 100 may obtain identification information corresponding to the hand object in the operation S1840. Specifically, the refrigerator 100 may obtain identification information corresponding to the hand object by identifying the shapes of the fingers, nails, the back of the hand, or the palm included in the hand object. Here, the identification information corresponding to the hand object may have been stored in advance. In other words, the refrigerator 100 may photograph hands for each user and register them, and store identification information corresponding to the hand objects in advance.

[0229] The refrigerator 100 may obtain feature information corresponding to the food object in the operation S1850. Specifically, the refrigerator 100 may obtain feature information corresponding to the food object by inputting the image into the first neural network model.

[0230] The refrigerator 100 may match the identification information corresponding to the hand object, the image, and the feature information corresponding to the food object, and store them in the food DB in the operation S1860. In other words, the refrigerator 100 may store together not only the image and the feature information corresponding to the food object, but also information on the user who put in the food.

[0231] FIG. 19 is a flow chart for illustrating a method of managing taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure.

[0232] Putting in of food may be generated in the operation S1910. Here, the refrigerator 100 may photograph an image regarding putting in of the food.

[0233] The refrigerator 100 may detect a hand object and a food object from the photographed image in the operation S1920.

[0234] The refrigerator 100 may separate an area corresponding to the detected hand object in the operation S1930. In other words, the refrigerator 100 may separate an area corresponding to the hand object from among the hand object and the food object detected from the photographed image.

[0235] The refrigerator 100 may obtain identification information corresponding to the hand object in the operation S1940. Specifically, the refrigerator 100 may obtain identification information corresponding to the hand object by identifying the shapes of the fingers, nails, the back of the hand, or the palm included in the hand object.

[0236] The refrigerator 100 may identify whether the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are matched in the operation S1950. In other words, the refrigerator 100 may determine whether the identification information corresponding to the hand object at the time of taking out of the food that is taken out and the identification information corresponding to the hand object at the time of putting in are identical.

[0237] If it is identified that the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are matched in the operation S1950-Y, the refrigerator 100 may store a single user putting in / taking out pattern in the operation S1960. Here, the single user putting in / taking out pattern may refer to a pattern by which one user puts in and takes out food.

[0238] Then, the refrigerator 100 may update the food DB in the operation S1990. In other words, the refrigerator 100 may remove food that was taken out from the food DB, and store identification information corresponding to a hand object related to putting in / taking out in the food history DB.

[0239] If it is identified that the identification information corresponding to the hand object at the time of taking out and the identification information corresponding to the hand object at the time of putting in are not matched in the operation S1950-N, the refrigerator 100 may provide a notification message notifying taking out by another user in the operation S1970. Specifically, if taking out is performed by another user, the refrigerator 100 may notify taking out by the another user through the display 123 or a user terminal.

[0240] The refrigerator 100 may store multi user putting in / taking out patterns in the operation S1980. Here, the multi user putting in / taking out patterns may refer to patterns by which a plurality of users put in and take out food.

[0241] Then, the refrigerator 100 may update the food DB in the operation S1990. In other words, the refrigerator 100 may remove food that was taken out from the food DB, and store identification information corresponding to a hand object related to putting in / taking out in the food history DB.

[0242] FIG. 20 is a diagram for illustrating an embodiment of managing putting in and taking out of food according to identification information corresponding to a hand object according to an embodiment of the disclosure. Meanwhile, each of the first to fifth images explained in FIG. 20 may be a plurality of image frames but not one image frame. However, for the convenience of explanation, explanation will be described by assuming one image frame.

[0243] The refrigerator 100 may identify putting in / taking out of food based on the first to fifth images (2010 to 2050) as illustrated in FIG. 20. Specifically, the refrigerator 100 may identify that “milk was taken out by the user A at 15:10 on Aug. 10, 2023” through the first image 2010. The refrigerator 100 may identify that “snacks were taken out by the user B at 15:21 on Aug. 10, 2023” through the second image 2020. The refrigerator 100 may identify that “milk was put in by the user B at 15:33 on Aug. 10, 2023” through the third image 2030. The refrigerator 100 may identify that “juice was taken out by the user C at 16:10 on Aug. 10, 2023” through the fourth image 2040. The refrigerator 100 may identify that “juice was taken out by the user C at 16:13 on Aug. 10, 2023” through the fifth image 2050.

[0244] As described above, the refrigerator 100 may identify and manage the types of food that is put in or taken out through a hand object and information on a user who puts in or takes out food.

[0245] As an example, if the same food is put in after being taken out by the same user within a predetermined time (e.g., 1 hour), the refrigerator 100 may identify that the food was put in again. However, if the same food is put in after being taken out by a different user even within the predetermined time (e.g., 1 hour), the refrigerator 100 may identify that new food was put in.

[0246] As an example, in case a user who put in food and a user who took out the same food are different, the refrigerator 100 may provide a notification to the users. Here, the refrigerator 100 may provide the notification through the display 123 or a user terminal.

[0247] As an example, the refrigerator 100 may extract food pattern information and store it. Here, the food pattern information may include information and pattern information for food that is put in or taken out in a specific time zone. The refrigerator 100 may utilize the food pattern information in update of the food list. Also, the refrigerator 100 may provide user pattern information to the user by using the food pattern information, and provide customized recommendation services for each user based on the user pattern information.

[0248] As an example, the refrigerator 100 may provide a sharing function among users who use the refrigerator 100. For example, in case the user A puts milk in the refrigerator 100, the refrigerator 100 may share the information through a message application, or share the information on the display 123 of the refrigerator 100. Also, the refrigerator 100 may track the user who takes out the milk put in by the user A, and the time zone of taking out.

[0249] FIG. 21 is a diagram for illustrating an object recognition model according to an embodiment of the disclosure.

[0250] An object recognition model 2120 is a model for recognizing a type of a food object, and it may be referred to as a second neural network model. In particular, the object recognition model 2120 may include a plurality of layers for recognizing a type of a food object. If a photographed image 2110 is input into the object recognition model 2120, the object recognition model 2120 may output the type of the food object included in the photographed image 2110, and a probability value thereof. As an example, the object recognition model 2120 may output an output value which is “tomatoes 85%” (2130). However, this is merely an example, and the object recognition model 2120 may output probability values for each of a plurality of candidate objects, and the refrigerator 100 may identify a candidate object having the highest probability value among the output probability values as the type of the food object. Also, the object recognition model 2120 may divide information on a type of a food object according to a plurality of levels and output it. For example, the object recognition model 2120 may output “major category (fresh food)—subcategory (vegetables)—detailed food name (asparagus)” or “major category (processed food)—subcategory (canned food)—detailed food name (tuna can)” as an output value.

[0251] The object recognition model 2120 according to an embodiment of the disclosure may be implemented as various neural network models such as a convolutional neural network (CNN), a Transformer, etc.

[0252] FIG. 22 is a flow chart for illustrating a method of managing putting in of food according to whether food was normally put in according to an embodiment of the disclosure.

[0253] The refrigerator 100 may recognize putting in of food in the operation S2210. Here, recognizing putting in of food may mean recognizing an operation of a food object of moving in the inside direction of the refrigerator 100.

[0254] The refrigerator 100 may identify whether the food was normally put in in the operation S2220. As an example, if a photographed image is input into the second neural network model, in case the type of the food object included in the photographed image cannot be identified (or, in case the probabilities that the food object may belong to specific types are all smaller than or equal to a predetermined value (e.g., 50%)), the refrigerator 100 may identify that the food was put in abnormally.

[0255] In case it was identified that the food was normally put in in the operation S2220-Y, the refrigerator 100 may add the food to the food list and output a success message in the operation S2230. In other words, the refrigerator 100 may add the food that was put in to the food list, and output a success message guiding that putting in of the food was recognized. Here, in the success message, information on the food that was put in and information on putting in may be included.

[0256] In case it was identified that the food was put in abnormally in the operation S2220-N, the refrigerator 100 may output a failure message in the operation S2240. In other words, the refrigerator 100 may output a failure message guiding failure to recognize putting in of the food. Here, in the failure message, a message inquiring about information on the food that was put in may be included.

[0257] FIG. 23 is a flow chart for illustrating a method of managing taking out of food according to whether food was normally taken out according to an embodiment of the disclosure.

[0258] The refrigerator 100 may recognize taking out of food in the operation S2310. Here, recognizing taking out of food may mean recognizing an operation of a food object of moving in the outside direction of the refrigerator 100.

[0259] The refrigerator 100 may identify whether the food was normally taken out in the operation S2320. As an example, if a photographed image is input into the second neural network model, in case the type of the food object included in the image cannot be identified, the refrigerator 100 may identify that the food was taken out abnormally.

[0260] In case it was identified that the food was taken out abnormally in the operation S2320-N, the refrigerator 100 may output a failure message in the operation S2330. In other words, the refrigerator 100 may determine that the food was taken out abnormally, and output a failure message. Here, abnormal taking out of the food may be an event wherein the type of the food that was taken out cannot be recognized. Here, in the failure message, a message inquiring about information on the food that was taken out may be included.

[0261] In case it was identified that the food was normally taken out in the operation S2320-Y, the refrigerator 100 may identify whether food matched with the recognized food that was taken out exists in the food DB in the operation S2340.

[0262] If it is identified that food matched with the recognized food that was taken out exists in the food DB in the operation S2340-Y, the refrigerator 100 may update the food DB, and output a success message in the operation S2350. In other words, the refrigerator 100 may remove or change the information on the food that was taken out in the food DB, and output a success message guiding that taking out of the food was recognized. Here, in the success message, information on the food that was taken out and information on taking out may be included.

[0263] If it is identified that food matched with the recognized food that was taken out does not exist in the food DB in the operation S2340-N, the refrigerator 100 may output a failure message guiding failure to recognize normal taking out of the food in the operation S2360. Here, in the failure message, a message inquiring about information on the food that was taken out may be included.

[0264] FIG. 24A and FIG. 24B are diagrams illustrating a UI providing information on food that is put in or taken out according to an embodiment of the disclosure.

[0265] When food is put in or taken out, the refrigerator 100 may update the food DB by reflecting information on the food that is put in or taken out. Then, the refrigerator 100 may provide a food list and information on the food that was put in or taken out based on the updated food DB. Here, at least one of the food list or the information on the food that was put in or taken out may be output whenever the door is closed after being opened, but this is merely an example, and it may be output based on a user input (e.g., a user touch).

[0266] As an example, as illustrated in FIG. 24A, the refrigerator 100 may provide a pop-up screen 2410 including information on the food that was put in and the food that was taken out. Here, the information on the food that was put in and the food that was taken out may be output on the display 123 of the refrigerator 100, but this is merely an example, and the information may be put on a display of a user terminal. Meanwhile, the feature that the refrigerator 100 provides the information on the food that was put in and the food that was taken out as the pop-up screen 2410 is merely an example, and the information may be provided in various forms (e.g., an image, a list, etc.).

[0267] Also, the refrigerator 100 may provide a voice assistant notification through the speaker 121. Alternatively, the refrigerator 100 may be connected with an external device (e.g., an AI speaker, etc.), and provide information on the food that was put in or taken out. For example, the refrigerator 100 may output a voice message which is “Cabbages, cheese, milk, and raw chicken were put on the shelf of the refrigerator. 2 L of milk, eggs, wine, and oranges were taken out” through the speaker 121 or a speaker of an external device.

[0268] Also, the refrigerator 100 may provide information for outputting a notification message (or a push message) to a user terminal. Here, the refrigerator 100 may provide information for outputting a notification message to a plurality of registered user terminals.

[0269] In addition, the refrigerator 100 may provide a notification message by being linked with a display device (e.g., a TV, etc.) among devices connected in a home.

[0270] Also, the refrigerator 100 may obtain recipe information based on information on the food that was taken out. For example, the refrigerator 100 may obtain recipe information related to the food that was taken out among the recipe information stored in the memory 160 in advance, and obtain recipe information related to the food that was taken out from an external server. Then, the refrigerator 100 may transmit the obtained recipe information to a device related to cooking of the food that was taken out (e.g., an oven, a microwave oven, a hood) among the devices connected in a home or a user terminal. By this, the user can check recipe information related to the food that was taken out more conveniently. In addition, the refrigerator 100 may identify a storage location of the food that was put in, and provide a UI screen 2420 including an inside image regarding the inside of the refrigerator 100 and the storage location of the food that was put in, as illustrated in FIG. 24B.

[0271] Also, the refrigerator 100 may provide information on whether the camera is being driven and recognition of putting in / taking out of food, etc. to the user as sound effects. Here, the sound effects may be output through the speaker 121 of the refrigerator 100, but this is merely an example, and the sound effects may be output through a speaker of an external device connected with the refrigerator 100. By this, the refrigerator 100 can immediately notify the user of information on whether the camera 110 operated normally, whether putting in / taking out of food was normally recognized, or was recognized incorrectly or was not recognized, etc. through sound effects, even if the history of putting in / taking out is not output on the display screen all the time.

[0272] Also, the refrigerator 100 may provide a sound effect regarding normal driving of the camera as a first sound effect, provide a sound effect regarding normal recognition of putting in of food as a second sound effect, provide a sound effect regarding normal recognition of taking out of food as a third sound effect, provide a sound effect regarding an error in putting in of food (non-recognition or misrecognition of food) as a fourth sound effect, and provide a sound effect regarding an error in taking out of food (non-recognition or misrecognition of food) as a fifth sound effect. Here, the first to fifth sound effects may be different from one another, but this is merely an example, and the second and third sound effects may be identical, and the fourth and fifth sound effects may also be identical. The first sound effect may be output on a time point when the camera started to be driven normally after the door is opened. The second sound effect or the third sound effect may be output whenever one piece of food is normally put in or taken out. The fourth sound effect or the fifth sound effect may be output whenever the user's hand / food was detected, but the type of the food cannot be identified.

[0273] FIG. 25A to FIG. 25C are diagrams for illustrating a candidate list for food that was taken out according to an embodiment of the disclosure.

[0274] When taking out of food is determined, the refrigerator 100 may not only match the food to the most similar food stored in the food DB, but may also output a candidate list including food having similar feature information. Here, the number of the pieces of food included in the candidate list may be changed, and the food may be output as much as the number designated by the user.

[0275] In particular, when the operation of managing putting in or taking out of food ends, the refrigerator 100 may provide the information on the food that was put in or taken out in a form of a list to the user. Then, if one of the food that was put in or taken out included in the list is selected, the refrigerator 100 may provide a candidate list for changing the selected food to another food.

[0276] For example, the refrigerator 100 may output a list 2510 including information on the food that was put in or taken out as illustrated in FIG. 25A. Here, if the first food 2511 is selected among the food included in the list 2510, the refrigerator 100 may display a candidate list 2520 including n pieces of candidate food 2521, 2522, 2523 having similar feature information to the feature information of the first food 2511 as illustrated in FIG. 25B. Here, the selected first food 2511 may be displayed to be distinguished from other food. Then, as illustrated in FIG. 25B, if the second candidate food 2522 in the candidate list is selected, the refrigerator 100 may change the first food 2511 to the second candidate food 2522 and display it, as illustrated in FIG. 25C. Further, in the candidate list, new fourth candidate food 2524 may be added. By this, the user can change the type of the recognized food that was taken out to another type, and thus user convenience can be increased.

[0277] FIG. 26A and FIG. 26B are diagrams for illustrating an embodiment wherein a user corrects misrecognition of food that was taken out according to an embodiment of the disclosure.

[0278] The refrigerator 100 may output the history of food that was processed to have been taken out in the food DB per set period as a pop-up screen. In particular, the refrigerator 100 may store the history of the food that was processed to have been taken out in the food history DB, and output it per set period. Here, the set period may be, for example, a week, but this is merely an example, and it may be changed by a user input.

[0279] As an example, as illustrated in FIG. 26A, the refrigerator 100 may display a pop-up screen 2610 including information on the food that was processed to have been taken out.

[0280] When the pop-up screen 2610 is output, the user may check the history of the food that was processed to have been taken out, and if there is food that was not actually taken out, but was processed to have been taken out, the user may add the food to the food DB again. For example, if the first food (e.g., eggs) and the second food (e.g., carrots) are selected on the pop-up screen 2610, and then the “add to the food list” button is selected, the refrigerator 100 may add the first food and the second food to the food DB again, and display a guide message 2620 which is “Added to the food list,” as illustrated in FIG. 26B.

[0281] As an example, the refrigerator 100 may display a button for purchase of the food that was processed to have been taken out. When the button is selected, the refrigerator 100 may connect to a website for purchasing the food that was processed to have been taken out and display a food purchase page. By this, the user can purchase the food that was processed to have been taken out more conveniently.

[0282] Meanwhile, the orders in the follow charts in the aforementioned various embodiments are merely examples, and it is obvious that the orders of each operation in the flow charts can be changed, and the orders in the flow charts can also be performed simultaneously.

[0283] Meanwhile, the processor 170 according to an embodiment of the disclosure performs control to process input data according to predefined operation rules or an artificial intelligence model stored in the memory 160. The predefined operation rules or the artificial intelligence model are characterized in that they are made through learning.

[0284] Here, being made through learning means that a learning algorithm is applied to a plurality of training data, and predefined operation rules or an artificial intelligence model having desired characteristics are thereby made. Such learning may be performed in a device itself wherein artificial intelligence is performed according to the disclosure, or through a separate server / system.

[0285] An artificial intelligence model (e.g., first and second object detection networks) may consist of a plurality of neural network layers. The at least one layer has at least one weight value, and performs an operation of the layer through an operation result of the previous layer, and at least one defined operation. As examples of a neural network, there are a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann Machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a Transformer, but the neural network in the disclosure is not limited to the aforementioned examples excluding specified cases.

[0286] A learning algorithm is a method of training a specific subject device by using a plurality of training data and thereby making the specific subject device make a decision or make prediction by itself. As examples of learning algorithms, there are supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but learning algorithms in the disclosure are not limited to the aforementioned examples excluding specified cases.

[0287] Meanwhile, methods according to the various embodiments of the disclosure may be provided while being included in a computer program product. A computer program product refers to a product, and it can be traded between a seller and a buyer. A computer program product can be distributed in the form of a storage medium that is readable by machines (e.g.: compact disc read only memory (CD-ROM)), or distributed on-line (e.g.: download or upload) through an application store (e.g.: Play Store™), or directly between two user devices (e.g.: smartphones). In the case of on-line distribution, at least a portion of a computer program product (e.g.: a downloadable app) may be stored in a storage medium readable by machines such as the server of the manufacturer, the server of the application store, and the memory of the relay server at least temporarily, or may be generated temporarily.

[0288] Also, the method according to the various embodiments of the disclosure may be implemented as software including instructions stored in machine-readable storage media, which can be read by machines (e.g.: computers). The machines refer to apparatuses that call instructions stored in a storage medium, and can operate according to the called instructions, and the apparatuses may include an electronic apparatus according to the aforementioned embodiments (e.g.: a refrigerator).

[0289] Meanwhile, a storage medium that is readable by machines may be provided in the form of a non-transitory storage medium. Here, the term ‘a non-transitory storage medium’ only means that the device is a tangible device, and does not include a signal (e.g.: an electromagnetic wave), and the term does not distinguish a case wherein data is stored semi-permanently in a storage medium and a case wherein data is stored temporarily. For example, ‘a non-transitory storage medium’ may include a buffer wherein data is temporarily stored.

[0290] In case an instruction is executed by a processor, the processor may perform a function corresponding to the instruction by itself, or by using other components under its control. An instruction may include a code that is generated or executed by a compiler or an interpreter.

[0291] Also, while preferred embodiments of the disclosure have been shown and described, the disclosure is not limited to the aforementioned specific embodiments, and it is apparent that various modifications may be made by those having ordinary skill in the technical field to which the disclosure belongs, without departing from the gist of the disclosure as claimed by the appended claims. Further, it is intended that such modifications are not to be interpreted independently from the technical idea or prospect of the disclosure.

Examples

Embodiment Construction

[0052]Various embodiments of the disclosure and the terms used in the embodiments are not for limiting the technological characteristics described in the disclosure to specific embodiments, but they should be interpreted to include various modifications, equivalents, or alternatives of the embodiments.

[0053]Also, with respect to the detailed description of the drawings, similar or related components may be designated by similar reference numerals.

[0054]In addition, a singular form of a noun corresponding to an item may include one of the item or a plurality of the items, unless instructed obviously differently in the related context.

[0055]Further, in the disclosure, each of the phrases “A or B,”“at least one of A and B,”“at least one of A or B,”“A, B, or C,”“at least one of A, B, and C,” and “at least one of A, B, or C” and the like may include any one of the items listed together with the phrase among the phrases, or all possible combinations of the listed items.

[0056]Also, the ter...

Claims

1. A refrigerator comprising:a body including a storage chamber;a door, which is rotatably coupled to the body to open and close the storage chamber, including a door bin;a camera, arrangeable in the body, configured to photograph an inside of the body and an inside of the door;a memory to store at least one instruction; anda processor configured to:based on detecting a trigger signal, obtain an image of at least a portion of the inside of the body and at least a portion of the door that is photographed through the camera while the camera is arranged in the body,detect a food object included in the obtained image,track the food object and identify whether the food object is put in or taken out of the refrigerator,based on identifying that the food object is put in the refrigerator, obtain feature information corresponding to the food object by inputting the image into a trained neural network model, andmatch the image and the obtained feature information, and store the image and the obtained feature information based on the match in a food database.

2. The refrigerator of claim 1,wherein the processor is configured to:obtain information associated with the food object by inputting the image into a trained second neural network model, andstore the information associated with the food object in the food database together with the image and the obtained feature information, andthe information associated with the food object comprises:at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

3. The refrigerator of claim 1,wherein the processor is configured to:crop an area including the food object from the image, andobtain the feature information corresponding to the food object by inputting an area where the food object is included into the trained neural network model.

4. The refrigerator of claim 1, wherein the obtained feature information is first feature information, andthe processor is configured to:based on identifying that the food object is taken out, identify the second feature information which was obtained when the food object was put in in the food database, andbased on the identifying the second feature information, delete the second feature information from the food database.

5. The refrigerator of claim 4, wherein the processor is configured to:based on the second feature information not being identified, identify a candidate list corresponding to the food object, andbased on identifying the candidate list, provide user interface to receive user's selection for deleting at least one candidate from the food database,wherein the candidate list comprises:at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

6. The refrigerator of claim 4, wherein the processor is configured to:based on identifying that the food object was taken out and not is put in within a predetermined time, delete the second feature information from the food database7. The refrigerator of claim 1, wherein the food object comprises a first food object and a second food object, andthe processor is configured to:identify that the second food object is put in the refrigerator,based on identifying that the second food object is put in, identify whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, andbased on identifying that the first food object and the second food object are matched, determine occurrence of an event where the first food object is put in again, and store information on the event in the food database.

8. The refrigerator of claim 7,wherein the processor is configured to:based on identifying that the first food object matching the second food object does not exist, store the image and the obtained feature information as new food object.

9. The refrigerator of claim 1,wherein the processor is configured to:based on identifying that the food object is put in, identify whether the feature information corresponding to the food object is obtainable,based on identifying the feature information corresponding to the food object is unobtainable, obtain information associated with an area corresponding to the food object from the image, andmatch the image and the information associated with the area corresponding to the food object, and store the image and the information with the area in the food database.

10. The refrigerator of claim 1,wherein the processor is configured to:detect a hand object from the image,obtain identification information associated with the detected hand object based on a plurality of pre-stored hand objects, andmatch the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database.

11. A control method for a refrigerator, the control method comprising:based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body;detecting a food object included in the obtained image;tracking the food object and identifying whether the food object is put in or taken out of the refrigerator;based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; andmatching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.

12. The control method of claim 11,wherein the control method further comprises:obtaining information associated with the food object by inputting the image into a trained second neural network model, andthe storing comprises:storing the information associated with the food in the food database together with the image and the obtained feature information, andthe information associated with the food object comprises:at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object.

13. The control method of claim 11,wherein the control method comprises:cropping an area including the food object from the image, andthe obtaining the feature information comprises:obtaining the feature information corresponding to the food object by inputting an area wherein the food object is included into the trained neural network model.

14. The control method of claim 11, wherein the obtained feature information is first feature information, andwherein the control method comprises:based on identifying that the food object is taken out, identifying the second feature information which was obtained when the food object was put in in the food database, andbased on the identifying the second feature information, deleting the second feature information from the food database.

15. The control method of claim 14, wherein the control method comprises:based on the second feature information not being identified, identifying a candidate list corresponding to the food object, andbased on identifying the candidate list, providing user interface to receive user's selection for deleting at least one candidate from the food database,wherein the candidate list comprises:at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.

16. The control method of claim 14, wherein the control method comprises:based on identifying that the food object was taken out and not is put in within a predetermined time, deleting the second feature information from the food database.

17. The control method of claim 11, wherein the food object comprises a first food object and a second food object, andwherein the control method comprises:identifying that the second food object is put in the refrigerator,based on identifying that the second food object is put in, identifying whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, andbased on identifying that the first food object and the second food object are matched, determining occurrence of an event where the first food object is put in again, and storing information on the event in the food database.

18. The control method of claim 17, wherein the control method comprises:based on identifying that the first food object matching the second food object does not exist, storing the image and the obtained feature information as new food object.

19. The control method of claim 11, wherein the control method comprisesbased on identifying that the food object is put in, identifying whether the feature information corresponding to the food object is obtainable,based on identifying the feature information corresponding to the food object is unobtainable, obtaining information associated with an area corresponding to the food object from the image, andmatching the image and the information associated with the area corresponding to the food object, and storing the image and the information with the area in the food database.

20. One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of a refrigerator individually or collectively, cause the refrigerator to perform operations, the operations comprising:based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body;detecting a food object included in the obtained image;tracking the food object and identifying whether the food object is put in or taken out of the refrigerator;based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; andmatching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.

Images