Refrigerator and control method therefor

Abstract

Provided are a refrigerator and a control method therefor. The refrigerator of the present disclosure includes: a body including a storage chamber; a door rotatably coupled to the body to open and close the storage chamber and including a door bin; a camera positioned in the body and photographing the inside of the body and the inside of the door; a memory for storing at least one instruction; and a processor which acquires an image by photographing at least a portion of the inside of the body and the door through the camera when a trigger signal is detected, detects a first food object included in the acquired image, tracks the first food object and identifies whether the first food object is put in or taken out of the refrigerator, acquires first feature information corresponding to the first food object by inputting the image to a trained first neural network model when it is identified that the first food object is put in, and matches the image and the acquired first feature information and stores the image and the acquired first feature information in a food database.

Description

Refrigerator and method of controlling the same

[0001] The present disclosure relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof capable of managing the entry and exit of food objects by photographing the inside of the refrigerator.

[0002] In general, a refrigerator is a home appliance that has a storage compartment for storing food and a cold air supply device that supplies cold air to the storage compartment, allowing food to be kept fresh for a long time.

[0003] In particular, recent refrigerators feature cameras embedded in the main body, including the storage compartment, to capture images of the interior. Based on the captured images, the refrigerator acquires information about the food currently stored in the refrigerator and provides this information to the user.

[0004] However, conventional systems have employed multiple cameras to obtain accurate information about food. However, refrigerators face limitations on the number and location of cameras, including locations that minimize food contamination and are not affected by temperature. Furthermore, this increases material costs.

[0005] Therefore, a method is required to accurately manage the entry and exit of food objects using a minimum number of cameras.

[0006] According to one embodiment of the present disclosure, a refrigerator comprises: a main body including a storage compartment; a door rotatably coupled to the main body to open and close the storage compartment and including a door bin; a camera positioned in the main body and photographing the inside of the main body and the inside of the door; a memory storing at least one instruction; and a processor configured to, when a trigger signal is detected, photograph at least a portion of the inside of the main body and the door through the camera to obtain an image, detect a first food object included in the obtained image, track the first food object to identify whether the first food object is being stored in or shipped from the refrigerator, and, when the first food object is identified as being stored, input the image into a trained first neural network model to obtain first feature information corresponding to the first food object, and match the image and the obtained first feature information to store the matched image in a food database.

[0007] The processor obtains information about the first food object by inputting the image into a learned second neural network model, and stores the information about the food object together with the image and the obtained first feature information in the food database, and the information about the first food object may include at least one of a type of the first food object, a product name of the first food object, a manufacturer of the first food object, and a capacity of the first food object.

[0008] The processor can crop an area including the first food object from the image and input the area including the first food object into the learned first neural network model to obtain first feature information corresponding to the first food object.

[0009] The processor, when identified as shipping the first grocery object, inputs the image into the learned first neural network model to obtain second feature information corresponding to the first grocery object, identifies whether there is a grocery item matching the second feature information among the grocery items stored in the grocery database, and when it is identified that there is a grocery item matching the second feature information, deletes the grocery item matching the second feature information from the grocery database.

[0010] The processor may identify whether a second grocery object has been received within a preset time after the first grocery object has been identified as being shipped, and if the second grocery object has been identified as being received within the preset time, identify whether the first grocery object and the second grocery object match, and if the first grocery object and the second grocery object have been identified as being matched, identify a restocking event of the first grocery object, and store information related to the restocking event in the grocery database.

[0011] The processor may identify whether there is a food item corresponding to the first food item object among the food items stored in the food database, if it is determined that the second food item object has not been received within the preset time or if it is determined that the first food item object and the second food item object do not match, and if it is determined that there is a food item corresponding to the first food item object, it may delete the food item corresponding to the first food item object from the food database.

[0012] The processor, if it is determined that there is no food corresponding to the first food object, provides a candidate list for the first food object, and the candidate list may include at least one of n food objects corresponding to feature information most similar to third feature information corresponding to the first food object, an unspecified packaging object, and a recently received food object.

[0013] If the processor identifies that there is no food corresponding to the first food object, it can identify whether there is an unspecified packaging object that was delivered before the preset time, and if it identifies that there is an unspecified packaging object, it can determine that the first food object and the unspecified packaging object correspond and update the food database.

[0014] The processor, when it is identified that the first grocery object is received, identifies whether acquisition of first characteristic information corresponding to the first grocery object is possible, and when it is identified that acquisition of first characteristic information corresponding to the first grocery object is impossible, it acquires information about an area corresponding to the first grocery object from the image, and matches the image with information about an area corresponding to the first grocery object and stores the results in the grocery database.

[0015] The processor can detect a hand object from the image, obtain identification information for the detected hand object based on a plurality of previously stored hand objects, and match the identification information for the hand object, the image, and the first feature information to store the result in the food database.

[0016] Meanwhile, according to one embodiment of the present disclosure, a control method for a refrigerator including a main body including a storage compartment, a door rotatably coupled to the main body to open and close the storage compartment and including a door bin, and a camera positioned on the main body and photographing the inside of the main body and the inside of the door, comprises the steps of: when a trigger signal is detected, photographing at least a portion of the inside of the main body and the door through the camera to obtain an image; detecting a first food object included in the obtained image; tracking the first food object to identify whether the first food object is being stored in or shipped from the refrigerator; when it is identified that the first food object is being stored, inputting the image into a trained first neural network model to obtain first feature information corresponding to the first food object; and matching the image and the obtained first feature information to store the matched image in a food database.

[0017] The control method further includes a step of inputting the image into a learned second neural network model to obtain information about the first food object; and the storing step stores the information about the food object together with the image and the obtained first feature information in the food database, and the information about the first food object includes at least one of a type of the first food object, a product name of the first food object, a manufacturer of the first food object, and a capacity of the first food object.

[0018] The above control method includes a step of cropping an area including the first food object from the image; and the step of obtaining the first feature information may input the area including the first food object into the learned first neural network model to obtain first feature information corresponding to the first food object.

[0019] The control method may include, when the first grocery object is identified as being shipped, a step of inputting the image into the learned first neural network model to obtain second feature information corresponding to the first grocery object; a step of identifying whether a grocery item matching the second feature information exists among the grocery items stored in the grocery database; and a step of deleting the grocery item matching the second feature information from the grocery database when it is identified that a grocery item matching the second feature information exists.

[0020] The control method may include: a step of identifying whether a second grocery object has been received within a preset time after the first grocery object has been identified as being shipped; a step of identifying whether the first grocery object and the second grocery object match if the second grocery object has been identified as being received within the preset time; and a step of identifying a restocking event of the first grocery object and storing information related to the restocking event in the grocery database if the first grocery object and the second grocery object have been identified as being matched.

[0021] The control method may include a step of identifying whether there is a food corresponding to the first food object among the food stored in the food database if it is determined that the second food object has not been received within the preset time or if it is determined that the first food object and the second food object do not match; and a step of deleting the food corresponding to the first food object from the food database if it is determined that there is a food corresponding to the first food object.

[0022] The above control method includes a step of providing a candidate list for the first grocery object when it is determined that no grocery corresponding to the first grocery object exists; wherein the candidate list may include at least one of n grocery objects corresponding to feature information most similar to third feature information corresponding to the first grocery object, an unspecified packaging object, and a recently received grocery object.

[0023] The above control method may include, if it is determined that there is no food corresponding to the first food object, a step of identifying whether there is an unspecified packaging object that was delivered before the preset time; and, if it is determined that there is an unspecified packaging object, a step of updating the food database by determining that the first food object and the unspecified packaging object correspond.

[0024] The control method may include, when the first grocery object is identified as being received, a step of identifying whether acquisition of first characteristic information corresponding to the first grocery object is possible; when acquisition of the first characteristic information corresponding to the first grocery object is identified as impossible, a step of acquiring information about an area corresponding to the first grocery object from the image; and a step of matching the image with information about an area corresponding to the first grocery object and storing the result in the grocery database.

[0025] The control method includes a step of detecting a hand object from the image; and a step of obtaining identification information for the detected hand object based on a plurality of previously stored hand objects; and the step of storing may store the identification information for the hand object, the image, and the first characteristic information in the food database.

[0026] FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to one embodiment of the present disclosure;

[0027] FIGS. 2A to 2E are drawings for explaining the internal configuration of a refrigerator according to one embodiment of the present disclosure.

[0028] FIG. 3 is a flowchart illustrating a method for managing the receipt and delivery of groceries according to one embodiment of the present disclosure;

[0029] FIG. 4 is a diagram illustrating a method for tracking a grocery object and a hand object according to one embodiment of the present disclosure;

[0030] FIG. 5 is a diagram illustrating a grocery list according to one embodiment of the present disclosure;

[0031] FIG. 6 is a diagram illustrating a UI for modifying groceries included in a grocery list according to one embodiment of the present disclosure;

[0032] FIG. 7 and FIG. 8 are drawings for explaining a grocery history list according to one embodiment of the present disclosure;

[0033] FIG. 9 is a flowchart illustrating a method for managing the shipment of groceries according to one embodiment of the present disclosure;

[0034] FIG. 10 is a drawing for explaining an embodiment in which groceries are restocked within a preset time according to one embodiment of the present disclosure;

[0035] FIG. 11 is a flowchart illustrating a method for managing the shipment of groceries according to one embodiment of the present disclosure;

[0036] FIG. 12 is a diagram for explaining an embodiment of managing an unspecified packaging object according to one embodiment of the present disclosure;

[0037] FIG. 13 is a flowchart illustrating a method for managing the shipment of groceries according to one embodiment of the present disclosure;

[0038] FIG. 14 is a flowchart illustrating a method for managing the receipt and delivery of bundled groceries according to one embodiment of the present disclosure;

[0039] FIG. 15 is a drawing for explaining an embodiment of managing the receipt and delivery of bundled groceries according to one embodiment of the present disclosure;

[0040] FIGS. 16 to 17b are drawings for explaining a method for managing the receipt of groceries depending on whether specific information is obtained, according to one embodiment of the present disclosure.

[0041] FIG. 18 is a flowchart illustrating a method for managing the receipt of groceries according to identification information corresponding to a hand object according to one embodiment of the present disclosure;

[0042] FIG. 19 is a flowchart illustrating a method for managing the delivery of groceries according to identification information corresponding to a hand object according to one embodiment of the present disclosure;

[0043] FIG. 20 is a diagram for explaining an embodiment of managing the receipt and delivery of groceries according to identification information corresponding to a hand object according to one embodiment of the present disclosure;

[0044] FIG. 21 is a drawing for explaining an object recognition model according to one embodiment of the present disclosure;

[0045] FIG. 22 is a flowchart for explaining a method for managing the receipt of groceries according to whether the groceries are normally received, according to one embodiment of the present disclosure.

[0046] FIG. 23 is a flowchart for explaining a method for managing the shipment of groceries based on whether the groceries are normally shipped, according to one embodiment of the present disclosure.

[0047] FIG. 24a and FIG. 24b are diagrams illustrating a UI that provides information on groceries being received or shipped, according to one embodiment of the present disclosure.

[0048] FIGS. 25A to 25C are drawings for explaining a candidate list for shipped groceries according to one embodiment of the present disclosure, and

[0049] FIGS. 26A and 26B are drawings for explaining an embodiment in which a user corrects a misrecognition of a shipped food product according to one embodiment of the present disclosure.

[0050] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather include various modifications, equivalents, or substitutes of the embodiments.

[0051] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0052] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0053] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0054] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0055] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0056] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0057] Terms such as "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0058] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0059] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0060] A refrigerator according to one embodiment may include a body.

[0061] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0062] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0063] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0064] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0065] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0066] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0067] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0068] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0069] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0070] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0071] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0072] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door bin for storing items. The door bin may be referred to as a door basket or door bin.

[0073] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0074] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0075] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0076] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0077] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment through heat generation and cooling through the Peltier effect.

[0078] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0079] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0080] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0081] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0082] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0083] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0084] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0085] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0086] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0087] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0088] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0089] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) to which the refrigerator or user device is connected to the wide area network (WAN) to which the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0090] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0091] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0092] Meanwhile, in the present disclosure, the meaning of the refrigerator (100) “providing” information may include not only displaying information through a display (123) included in the refrigerator (100), but also transmitting information to a user terminal that is in communication with the refrigerator (100) and displaying the information through a display of the user terminal.

[0093] Hereinafter, a refrigerator according to various embodiments will be described in detail with reference to the attached drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the present disclosure. As illustrated in FIG. 1, a refrigerator (100) may include a camera (110), an output unit (120), a communication interface (130), a microphone (140), a sensor (150), a memory (160), and a processor (170). The refrigerator (100) may be a device for storing food or medicine at a preset temperature to prevent them from being chilled or spoiled. The refrigerator (100) according to an embodiment of the present disclosure is illustrated in the shape of a typical household refrigerator, but is not limited thereto, and may be a kimchi refrigerator, a liquor refrigerator, a cosmetics refrigerator, a freezer, etc.

[0094] The camera (110) is a configuration for capturing a subject to generate a captured image, wherein the captured image may include both a moving image and a still image. Meanwhile, the "image" of the present disclosure may be a concept including both an image output on the display (123) and an image frame captured by the camera (110). In addition, the captured image may include at least one object. According to one embodiment of the present disclosure, the "object" is a configuration included in an image captured by the camera, and may include a food object, a hand object, a face object, a kitchenware object, etc.

[0095] In particular, the camera (110) can capture images of the storage compartment inside the main body (230) of the refrigerator (100) and the door bin (or door basket, pantry) area of ​​the doors (210, 220). The camera (110) can be installed in at least one of the upper area, the lower area, and the side area inside the main body (230) to capture images of the inside of the main body (230) and the door bin area of ​​the doors (210, 220). In addition, the camera (110) can be installed on the outside of the refrigerator (100) to capture images of the outside of the refrigerator (100). That is, the camera (110) can be implemented not only as a single camera, but also as a plurality of cameras depending on the embodiment. This will be described in more detail with reference to FIGS. 2A to 2E.

[0096] FIG. 2A is a drawing illustrating a refrigerator according to one embodiment of the present disclosure when the door is open. The refrigerator (100) of FIG. 2A is illustrated as having a plurality of doors (210, 220) on both upper sides, but is not limited thereto, and depending on the arrangement of the doors (210, 220), storage compartments (230), etc., the refrigerator (100) may be implemented as a French door type, a side-by-side type, etc.

[0097] A refrigerator (100) may be provided with doors (210, 220) on both upper sides. In addition, the refrigerator (100) may include a display (123) on at least one of the doors (210, 220) on both upper sides. In this case, not only information about food stored in the refrigerator (100) but also various information (e.g., event information received from the outside, alarm information, recipe information, etc.) may be provided on the display (123).

[0098] The refrigerator (100) may further include a storage compartment (230), as illustrated in FIG. 2A. The storage compartment (230) may be opened by a door (210, 220) that opens and closes, and may store water, beverages, and refrigerated or frozen foods. At this time, the storage compartment (230) may include a plurality of storage spaces and storage areas. The storage compartment (230) may be partitioned by partitions arranged within the main body. The storage compartment may be divided into a freezer compartment arranged at the bottom of the refrigerator (100) and a refrigerator compartment arranged at the top. However, the arrangement of the freezer compartment and the refrigerator compartment is not limited thereto, and they may be arranged in an interchangeable position.

[0099] The door (210, 220) can be rotated at an angle set by a hinge (e.g., less than 300°) to open and close a portion of the front of the storage compartment (230).

[0100] At this time, the second door (220) among the plurality of doors (210, 220) may include a display (123) that displays functions and settings of the refrigerator (100) on its surface and can be changed by user input (e.g., selection of a touch or button). In addition, at least some of the plurality of doors (210, 220) may further include a dispenser that provides water, ice, or carbonated water and / or a grippable handle.

[0101] Meanwhile, as illustrated in FIG. 2a, the refrigerator (100) may include a camera (110) in the upper area of ​​the main body to photograph at least a portion of the storage compartment (230) and the door bin of the door (210, 220).

[0102] FIG. 2b is a top perspective view of a refrigerator (100), and the camera (110) can be positioned in the upper central area of ​​the main body (particularly, the top table) to capture images of both the door bins of the first door (210) and the second door (220), as shown in FIG. 2b. FIG. 2c is a cross-sectional view taken along line A-A' of the drawing of FIG. 2b, and the camera (110) can be positioned to face downward at a preset angle (e.g., 30 degrees) to capture images of at least a portion of the storage compartment (230) and the door bins of the doors (210, 220), as shown in FIG. 2c.

[0103] FIG. 2d and FIG. 2e are upper perspective views of the refrigerator (100), and are drawings illustrating a shooting range (240-1, 240-2) captured by the camera (110). FIG. 2d is a drawing illustrating a shooting range (240-1) captured by the camera (110) when the door (210, 220) of the refrigerator (100) is fully open, and FIG. 2e is a drawing illustrating a shooting range (240-2) captured by the camera (110) when the door (210, 220) of the refrigerator (100) satisfies a preset condition. At this time, the “preset condition” is a condition for obtaining an optimal door bin image, and at this time, the “optimal door bin image” may be an image that satisfies an optimal condition for identifying information on food contained in the door bin.

[0104] Additionally, the camera (110) can provide images captured by the processor (170) to manage the receipt and delivery of food.

[0105] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide angle of view, but is not limited thereto.

[0106] Meanwhile, in FIGS. 2A to 2E, the camera (110) is described as being located at the top inside the main body, but this is only one embodiment, and it is obvious that multiple cameras may be provided in other areas inside the main body (e.g., rear area, bottom area, side area, etc.).

[0107] The output unit (120) can provide various feedbacks. In particular, the output unit (120) may be equipped with a speaker (121), an LED (Light Emitting Diode) (122), a display (123), etc., as illustrated in FIG. 1. However, this is only one embodiment, and other output units (e.g., a haptic providing device, etc.) may be further included.

[0108] At this time, the speaker (121) may be installed inside or outside the refrigerator (100) to provide various auditory feedback through audio. The LED (122) may be installed inside the storage compartment (230) or door (210, 220) inside the refrigerator (100) to provide various visual feedback through indicators of a specific shape (e.g., arrows, etc.) and blinking. The display (123) may be located in at least some areas among the plurality of doors (210, 220) to provide various visual feedback to the user.

[0109] In particular, the output unit (120) can output information indicating that groceries have been stored in and out of the refrigerator, information on groceries stored in and out, a grocery list, a candidate list, etc. This will be described in detail later with reference to the drawings.

[0110] The communication interface (130) can communicate with an external server or an external terminal device. In particular, the communication interface (130) can transmit an image including food to an external server to obtain information about the food, and can receive information about the food from the external server. In addition, the communication interface (130) can transmit information about the food and information about the storage location of the food to a user terminal, and can receive control commands from the user terminal. In this case, the communication interface (130) can communicate directly with the user terminal, but this is only one embodiment, and it is obvious that the communication interface (130) can communicate with an external user terminal through a server.

[0111] In particular, the communication interface (130) can communicate with various external devices using various wireless communication technologies or mobile communication technologies. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared Data Association (IrDA), or near field communication (NFC), and such mobile communication technologies may include, for example, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, and such.

[0112] The microphone (140) is a component that acquires an audio signal and converts it into an electrical signal, and may be installed inside or outside the refrigerator (100). In particular, the microphone (140) may receive an audio signal including a user's voice. In this case, the user's voice may include information about receipt or shipment (hereinafter referred to as "receipt / shipment") and information about food (e.g., type of food, expiration date of food, etc.).

[0113] The sensor (150) can detect the operating status of the refrigerator (100) (e.g., power or temperature) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. In particular, the processor (170) can measure the temperature of each of the plurality of storage compartments of the refrigerator (100) through the sensing value acquired through the sensor (150). Alternatively, the processor (170) can recognize that a user is approaching through the sensor (150) and control the camera (110) to be in a ready state. Alternatively, the processor (170) can detect the opening of the door through the sensor (150) and drive the camera (110).

[0114] The memory (160) can store an operating system (OS) for controlling the overall operation of the components of the refrigerator (100) and instructions or data related to the components of the refrigerator (100). In particular, the memory (160) can store various configurations for managing the entry and exit of food. In addition, the memory (160) can store a food database (DB) that stores information about food stored in the refrigerator (100) (e.g., type of food, capacity of food, expiration date of food, storage location of food, etc.).

[0115] Additionally, the memory (160) may, according to one embodiment, store a learned first neural network model (e.g., an object classification model, etc.) for obtaining feature information corresponding to food items received or shipped from the refrigerator (100). Alternatively, the memory (160) may store a learned second neural network model (e.g., an object recognition model, etc.) for recognizing food items received or shipped from the refrigerator (100).

[0116] Meanwhile, the memory (160) may be implemented as a non-volatile memory (e.g., hard disk, SSD (Solid state drive), flash memory), volatile memory (which may also include memory within the processor (170)), etc.

[0117] The processor (170) can control the refrigerator (100) according to at least one instruction stored in the memory (160).

[0118] In particular, the processor (170) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0119] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an AI-dedicated processor). For example, an operation for managing the receipt and delivery of groceries may be performed by a general-purpose processor such as a CPU, and an operation for acquiring feature information corresponding to a groceries object using a neural network model or an operation for recognizing groceries may be performed by an AI-dedicated processor such as an NPU.

[0120] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0121] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0122] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0123] According to one embodiment of the present disclosure, when a trigger signal is detected, the processor (170) captures an image by photographing at least a portion of the interior of the main body and the doors (210, 220) through the camera (110). At this time, the trigger signal is a signal for initiating a photographing operation of the camera (110) and may be generated by user proximity, door opening, user voice, etc. The processor (170) detects a first food object included in the captured image. The processor (170) tracks the first food object to identify whether the first food object is being stored in or taken out of the refrigerator. At this time, “storing” may be an operation of storing food in the refrigerator, and “taking out” may be an operation of taking food stored in the refrigerator out. If the first food object is identified as being stored, the processor (170) inputs the image into a trained first neural network model to obtain first feature information corresponding to the first food object. At this time, the first neural network model may be an object classification model for obtaining feature information corresponding to the food object. The processor (170) matches the image and the acquired first feature information and stores them in the food database.

[0124] In one embodiment, the processor (170) may input an image into a trained second neural network model to obtain information about a first food object. At this time, the second neural network model may be an object recognition model for recognizing the food object. The processor (170) may store information about the food object together with the image and the obtained first feature information in a food database. At this time, the information about the first food object may include at least one of the type of the first food object, the product name of the first food object, the manufacturer of the first food object, and the capacity of the first food object.

[0125] In one embodiment, the processor (170) may crop an area including a first food object from an image. Then, the processor (170) may input the area including the first food object into a trained first neural network model to obtain first feature information corresponding to the first food object.

[0126] In one embodiment, if the first grocery object is identified as being shipped, the processor (170) may input the image into the trained first neural network model to obtain second feature information corresponding to the first grocery object. The processor (170) may identify whether there is a grocery item that matches the second feature information among the grocery items stored in the grocery database. If it is identified that there is a grocery item that matches the second feature information, the processor (170) may delete the grocery item that matches the second feature information from the grocery database.

[0127] In one embodiment, the processor (170) may identify whether a second grocery object has been received within a preset time period after the first grocery object has been identified as being shipped. If the second grocery object has been identified as being received within the preset time period, the processor (170) may identify whether the first grocery object and the second grocery object match. If the first grocery object and the second grocery object are identified as being matched, the processor (170) may identify a restocking event of the first grocery object and store information related to the restocking event in a grocery database.

[0128] In one embodiment, if it is determined that the second grocery object has not been received within the preset time or if it is determined that the first grocery object and the second grocery object do not match, the processor (170) may determine whether there is a grocery item corresponding to the first grocery object among the grocery items stored in the grocery database. If it is determined that there is a grocery item corresponding to the first grocery item, the processor (170) may delete the grocery item corresponding to the first grocery item from the grocery database.

[0129] In one embodiment, if it is determined that there is no food corresponding to the first food object, the processor (170) may provide a candidate list for the first food object. In this case, the candidate list may include at least one of n food objects corresponding to feature information most similar to the third feature information corresponding to the first food object, an unspecified packaging object, and a recently received food object.

[0130] In one embodiment, if it is determined that there is no food corresponding to the first food object, the processor (170) may determine whether there is an unspecified packaging object that was delivered before a preset time. If it is determined that there is an unspecified packaging object, the processor (170) may determine that the first food object and the unspecified packaging object correspond and update the food database.

[0131] In one embodiment, if it is identified that a first grocery object is being received, the processor (170) may determine whether acquisition of first characteristic information corresponding to the first grocery object is possible. If it is determined that acquisition of first characteristic information corresponding to the first grocery object is impossible, the processor (170) may acquire information about an area corresponding to the first grocery object from the image. The processor (170) may match the image with information about the area corresponding to the first grocery object and store the result in a grocery database.

[0132] In one embodiment, the processor (170) can detect a hand object from an image. The processor (170) can obtain identification information about the detected hand object based on a plurality of previously stored hand objects. The processor (170) can match the identification information about the hand object, the image, and the first feature information and store the result in a food database.

[0133] FIG. 3 is a flowchart illustrating a method for managing the receipt and delivery of groceries according to one embodiment of the present disclosure.

[0134] The refrigerator (100) can detect a trigger signal (S305). At this time, the trigger signal is a signal for driving the camera (110), and may be, for example, a trigger signal generated after a door opening is detected by a door open / close sensor, a trigger signal generated after a user's proximity is detected by a proximity sensor, or a trigger signal generated after a user's voice for driving the camera is input through a microphone (140). However, this is merely an example, and the trigger signal may of course be generated by other methods.

[0135] The refrigerator (100) can acquire an image by driving the camera (110) (S310). Driving the camera (110) is an operation that initiates the operation of the camera (110) to acquire an image, and can be expressed as activating the camera (110) or initiating the camera (100). At this time, the refrigerator (100) can acquire an image based on the initial settings of the camera (exposure time, FPS (frames per second), Gain, etc.). In addition, the refrigerator (100) can set a region of interest (ROI) in the image. At this time, the ROI region can be an area where the user's hand or the appearance of groceries being received or delivered can be confirmed.

[0136] In addition, when the camera (110) is located inside, constant lighting and environment can be maintained according to the lighting inside the refrigerator (100), but when the camera (110) is located outside, the shooting environment can change according to the external environment (e.g., external lighting, etc.). Accordingly, the refrigerator (100) can dynamically change the setting values ​​of the camera (110) based on the external environment (e.g., lighting direction, illuminance, etc.).

[0137] The refrigerator (100) can detect a food object (S315). At this time, the food object detected may be a moving food object (e.g., a food object held by a user's hand, a food object contained in a dishware such as a pot, a food object obscured by another food object, etc.). Specifically, the refrigerator (100) can recognize a user's motion using a learned neural network model. At this time, the learned neural network model is a motion recognition model trained to recognize the user's hand motion, and the user's hand motion can be learned by images captured by various external environments. The refrigerator (100) can detect a moving food object and a hand object by recognizing the user's hand motion.

[0138] According to one embodiment, the refrigerator (100) can crop an area including a food object detected from a captured image. The refrigerator (100) can input the area including the first food object into a trained first neural network model to obtain first feature information corresponding to the first food object. Alternatively, the refrigerator (100) can input the area including the first food object into a trained second neural network model to obtain information about the first food object.

[0139] Additionally, the refrigerator (100) can detect pantry objects, face objects, kitchenware objects, etc. in addition to hand objects and grocery objects.

[0140] The refrigerator (100) can track a food object (S320). Specifically, the refrigerator (100) can detect and track the food object for each image frame acquired by the camera (110). At this time, the refrigerator (100) can analyze the movement direction of the food object based on the location of the food object included in multiple image frames.

[0141] In one embodiment, as illustrated in FIG. 4, the refrigerator (100) can detect the movement direction of the food object (410) and the hand object (420) included in a plurality of image frames (400-1 to 400-6). At this time, the refrigerator (100) can detect the center point of the food object (410) included in the image (particularly, the cropped region) and detect the movement direction of the food object (410) according to the movement of the center point. In addition, the refrigerator (100) can determine in which area of ​​the refrigerator (100) the food object (410) is stored through the movement direction of the food object (410) detected in the first to fourth image frames (400-1 to 400-4). And, since the food object (410) detected in the first to fourth image frames (400-1 to 400-4) is not detected in the fifth and sixth image frames (400-5, 400-6), the refrigerator (100) can determine the event of food being received.

[0142] Meanwhile, in the above-described embodiment, it has been described that the food object (410) is detected and the food object is tracked, but this is only one embodiment, and the refrigerator (100) can track pantry objects, face objects, kitchenware objects, etc. in addition to hand objects and food objects.

[0143] The refrigerator (100) can determine whether a food object has been received or shipped (S325). In one embodiment, the refrigerator (100) can analyze a food object contained in a plurality of image frames to determine whether the food object has been received or shipped.

[0144] In one embodiment, if an event is detected where some of the plurality of image frames contain a grocery object and the remaining image frames do not contain the grocery object, the refrigerator (100) may determine that the grocery object has been received. Alternatively, if an event is detected where some of the plurality of image frames do not contain a grocery object and the remaining image frames contain the grocery object, the refrigerator (100) may determine that the grocery object has been shipped.

[0145] In one embodiment, the refrigerator (100) can determine whether a food object has been received or shipped based on the direction of movement of the food object included in a plurality of image frames. That is, if the food object included in a plurality of image frames is identified as moving toward the inside of the main body or toward the door bin, the refrigerator (100) can determine that the food object has been shipped. Alternatively, if the food object included in a plurality of image frames is identified as moving toward the outside of the refrigerator (100), the refrigerator (100) can determine that the food object has been shipped.

[0146] Meanwhile, the refrigerator (100) can acquire information about the food object by inputting the captured image (particularly, the area corresponding to the cropped food object) into the second neural network model (object recognition model) learned while detecting and tracking the food object. At this time, the information about the food object may include at least one of the type of the food object, the product name of the food object, the manufacturer of the first food object, and the capacity of the first food object, but is not limited thereto.

[0147] When the receipt of a food object is determined, the refrigerator (100) can input an image into a trained first neural network model (S330). At this time, the first neural network model may be a neural network model trained to input an image and acquire feature information corresponding to the food object. At this time, the first neural network model may be trained by a contrastive learning method. The contrastive learning method is a main learning method of self-supervised learning, and is a method of training such that feature information (or feature values) corresponding to similar images are located close to each other in a vector space, and feature information corresponding to different images are located far apart. By classifying objects through a neural network model trained by the contrastive learning method in this way, even if a portion of the food object is covered by a hand object, photographed from a different angle, deformed, or packaged in a different packaging method, accurate object classification can be achieved by the first neural network model.

[0148] In one embodiment, the first neural network model can be applied to learning by subdividing label units, and at this time, the label input by the user can be applied to learning of the first neural network model.

[0149] The refrigerator (100) can obtain first feature information from the first neural network model (S335). At this time, the first feature information may include feature values ​​corresponding to a food object. At this time, the first feature information may be used to identify food items to be shipped in the future.

[0150] The refrigerator (100) can match an image with first feature information (S340). That is, the refrigerator (100) can match and store an image (particularly, an area corresponding to a food object) with the first feature information. In addition, the refrigerator (100) can match and store information about a food object acquired by a second neural network model, together with the image and the first feature information. For example, the refrigerator (100) can store information about a food object, such as "the name of the food, the storage location of the food, the date of receipt of the food, the date of shipment, the type of the food, the barcode information of the food, and the text included in the food," together with the image and the first feature information.

[0151] The refrigerator (100) can update the food DB (S360). Specifically, the refrigerator (100) can match images, first feature information, and information about the food to the food DB and store them in the food DB. At this time, the refrigerator (100) can store the first feature information in a feature information list of the food DB. In particular, the refrigerator (100) can compare the similarity of the newly stored first feature information with other feature information in the feature information list, and sort the first feature information based on the comparison results to update the feature information list. In addition, the feature information list can sort feature information by considering user information such as user patterns, entry / exit times, etc. in addition to the feature information.

[0152] Once the delivery of a food object is determined, the refrigerator (100) can input an image into the trained first neural network model (S345). Since the first neural network model is identical to the previously described first neural network model, any redundant description will be omitted.

[0153] The refrigerator (100) can acquire second feature information from the first neural network model (S350). At this time, the second feature information may include feature values ​​corresponding to food objects. Furthermore, although the second feature information is described as different from the first feature information to distinguish between incoming and outgoing items, it may have the same value as the first feature information (or a value within a range determined to be the same food type).

[0154] The refrigerator (100) can identify food products that match the second feature information (S355). Specifically, the refrigerator (100) can identify food products that correspond to the feature information that is closest to the second feature information among the feature information stored in the food DB stored in the refrigerator (100). In addition, the refrigerator (100) can identify food products that correspond to the feature information that is closest to the second feature information as food products that match the second feature information. That is, since feature information represents a vector value in a vector space, the refrigerator (100) can obtain information on the similarity between the second feature information and the other feature information by calculating the distance between the second feature information and the other feature information.

[0155] Meanwhile, the refrigerator (100) can obtain food items matching the second feature information and the closest feature information, and can also obtain N candidate feature information items based on the similarity between the second feature information and other feature information, and obtain a candidate list including the N candidate feature information items. In this case, N items may be preset, but this is merely an example and may be changed by a user setting. The refrigerator (100) can receive a user command to select food items shipped to the user using the candidate list.

[0156] The refrigerator (100) can update the food DB (S360). Specifically, the refrigerator (100) can delete food items stored in the food DB that match the second characteristic information. Alternatively, the refrigerator (100) can change information about food items stored in the food DB that match the second characteristic information. For example, the refrigerator (100) can change the number of food items stored in the food DB.

[0157] The refrigerator (100) may provide a grocery list (S365). Specifically, the refrigerator (100) may provide the grocery list based on an updated grocery database. At this time, the grocery list may include information on grocery items stored in multiple areas of the refrigerator (100). In particular, the grocery list may be categorized into multiple areas of the refrigerator (100) and include information on grocery items stored in each of the multiple areas. For example, the grocery list may provide the user with information on grocery items stored in the left door bin, grocery items stored in the right door bin, grocery items stored in the first floor area inside the main body, grocery items stored in the second floor area inside the main body, and grocery items stored in the third floor area inside the main body. In addition, the grocery list may include information on grocery items that have been received or shipped. At this time, information on grocery items that have been received or shipped may include information on the time of receipt or shipment, the number of receipts or shipments, etc.

[0158] Meanwhile, according to the present disclosure, the grocery DB may include multiple types of DBs. For example, the first grocery DB may be a grocery DB that includes information about grocery items currently stored in the refrigerator (100). That is, when the first grocery item is received, information about the first grocery item may be added to the first grocery DB, and when the second grocery item is shipped, information about the second grocery item may be deleted from the first grocery DB. The first grocery DB may be used to manage grocery items currently stored in the refrigerator (100) and may be used to provide a grocery list. As another example, the second grocery DB may be a grocery DB that temporarily stores information about grocery items currently shipped from the refrigerator (100) for a preset period of time. In this case, the second grocery DB may be a grocery DB that is used to determine whether grocery items shipped from the refrigerator (100) are restocked. As another example, the third grocery DB may be a grocery DB that stores information about the history of grocery items being shipped and received. That is, the third grocery DB can accumulate and store information on grocery receipts and shipments by time period. The third grocery DB can store the entire grocery receipt and shipment history, allowing users to understand grocery receipt and shipment history. Meanwhile, the first grocery DB can be called the "grocery DB," the second grocery DB the "temporary storage grocery DB," and the third grocery DB the "grocery history DB."

[0159] In addition, the refrigerator (100) can provide the user with a delivery history using a food history database in addition to the food list. For example, as illustrated in FIG. 5, the refrigerator (100) can provide a list (510) including food items that have been delivered. In this case, the list (510) including food items that have been delivered can be sorted by food type, as illustrated in FIG. 5. However, this is merely an example, and it is of course possible to sort the list by delivery processing time.

[0160] The refrigerator (100) can update the food DB using the list (510) including the processed food items for shipment. Specifically, when a user inputs a command to select one food item from the list (510) including the processed food items for shipment as shown in FIG. 5 and modify the processing, the refrigerator (100) can provide a UI (610) for modifying the processing for the selected food item, as shown in FIG. 6. The UI (610) for modifying the processing for the shipment can provide a candidate list including a plurality of candidate food items and re-stocking items. When the user selects one candidate food item from the candidate list, the refrigerator (100) can modify the processing for the food item selected from the list (510) and perform the processing for the candidate food item selected from the UI (610). Accordingly, the refrigerator (100) can update the food item selected from the list (510) to be added to the food DB and can update the candidate food item selected from the UI (610) to be deleted from the food DB. Alternatively, when the “Re-Put” item is selected among the UI (610) for modifying the delivery process, the refrigerator (100) can update to add the groceries selected from the list (510) to the groceries DB.

[0161] As one embodiment of the present disclosure, FIGS. 7 and 8 are diagrams illustrating a grocery history list. In this case, the grocery history list can be generated based on a grocery history DB.

[0162] The first list (710) illustrated in FIG. 7 may be a list that sorts information on groceries stored in and out of the refrigerator (100) by date. At this time, the first list (710) may include, as illustrated in FIG. 7, the details of groceries stored in and out, information on groceries stored in and out, and images of groceries stored in and out. In addition, the first list (710) may be sorted (or classified) by various criteria. For example, the first list (710) may be sorted by the date of storage and outflow using the first UI element (720) of FIG. 7, and may be sorted by groceries name and groceries category. In addition, the first list (710) may be modified according to user input. For example, the refrigerator (100) may delete at least one selected groceries included in the first list (710) based on a user command input through the second UI element (730) of FIG. 7. At this time, the first and second UI elements (720, 730) can be displayed on the first list (710).

[0163] Additionally, the first list (710) may be updated to include only incoming or outgoing groceries. For example, when a user command is input to provide a second list containing only incoming groceries (e.g., a user command selecting the "IN" item (715) illustrated in FIG. 7), the refrigerator (100) may provide a second list (810) containing only incoming groceries among the groceries included in the first list (710), as illustrated in FIG. 8.

[0164] Hereinafter, various embodiments of the present disclosure will be described with reference to the drawings. Meanwhile, the method for identifying the receipt or shipment of food products through captured images has been described in detail in FIG. 3, and therefore, redundant descriptions thereof will be omitted in the embodiments below.

[0165] FIG. 9 is a flowchart illustrating a method for managing the shipment of groceries according to one embodiment of the present disclosure.

[0166] The refrigerator (100) can identify the delivery of food (S905). Specifically, the refrigerator (100) can determine the delivery of food detected from an image captured by the camera (110) and obtain characteristic information corresponding to the detected food. At this time, the refrigerator (100) can temporarily store information about the delivered food in a temporary food storage database (DB). After a preset period of time has elapsed, the refrigerator (100) can delete the information about the delivered food temporarily stored in the temporary food storage database (DB).

[0167] The refrigerator (100) can identify whether food has been received within a preset time period (S910). That is, the refrigerator (100) can identify whether food has been received within a preset time period starting from the time at which food shipment is identified (or the shipment time). For example, the refrigerator (100) can identify whether food has been received within a preset time period starting from the shipment time for each of a plurality of shipped foods. In this case, the preset time period may be a reference time period for determining whether to restock, and may be, for example, one hour. In this case, the preset time period may be set according to user input.

[0168] If it is determined that groceries have been received within a preset time period (S910-Y), the refrigerator (100) can identify whether there is groceries that have been shipped that match the groceries that have been received within the preset time period (S915). At this time, the refrigerator (100) can identify whether there is groceries that have been shipped that match the groceries that have been received within the preset time period based on the characteristic information corresponding to the groceries that have been received within the preset time period and the characteristic information corresponding to the groceries that have been shipped that are stored in the temporary storage groceries DB. That is, the refrigerator (100) can obtain (or calculate) the similarity (or distance) between the characteristic information corresponding to the groceries that have been received within the preset time period and the characteristic information corresponding to the groceries that have been shipped that are stored in the temporary storage groceries DB. In addition, the refrigerator (100) can identify whether there is groceries that have been shipped that match the groceries that have been received within the preset time period based on the obtained similarity (or distance).

[0169] If it is identified that there is a shipped groceries matching the groceries received within a preset time period (S915-Y), the refrigerator (100) can determine that the groceries have been restocked (S920). In this case, restocking refers to the restocking of groceries previously received in the refrigerator (100) within a preset time period after being shipped, and can be distinguished from the initial stocking.

[0170] The refrigerator (100) can update the food database (S925) based on the restocking of food. Specifically, if it is determined that food has been restocked, the refrigerator (100) can delete the information about the restocked food stored in the temporary food database and re-store the information about the restocked food in the food database. Furthermore, the refrigerator (100) can update the food database to maintain the arrival time of the restocked food as the initial arrival time, not the restocking arrival time, and to add restocking history (such as the shipping time and restocking time).

[0171] If it is determined that no groceries have been received within a preset time (S910-N) or if it is determined that no groceries have been shipped that match the groceries received within the preset time (915-N), the refrigerator (100) can compare the shipped groceries with the groceries DB (S930). That is, the refrigerator (100) can obtain (or calculate) a similarity (or distance) by comparing the characteristic information corresponding to the shipped groceries with the characteristic information of the groceries stored in the groceries DB.

[0172] The refrigerator (100) can identify whether there is food matching the shipped food (S935). Specifically, the refrigerator (100) can identify whether there is food having feature information whose similarity to the shipped food exceeds a threshold value (or whose distance is below a threshold value) with the feature information corresponding to the shipped food.

[0173] If it is identified that there is food matching the shipped food (S935-Y), the refrigerator (100) can determine whether the food has been shipped (S940). That is, the refrigerator (100) can determine whether the food has been shipped in step S905.

[0174] The refrigerator (100) can update the food DB according to the delivery of food (S940). That is, the refrigerator (100) can delete or change information about food items that match the delivered food items among the food items stored in the food DB.

[0175] If it is determined that no groceries match the shipped groceries (S395-N), the refrigerator (100) may provide a candidate list (S950). The candidate list is provided to the user to identify the shipped groceries, and may include at least one of n groceries, unspecified packaged groceries, and recently received groceries that correspond to the most similar characteristic information to the characteristic information corresponding to the shipped groceries.

[0176] The refrigerator (100) can perform a delivery process based on user input (S955). Specifically, when a user input is received to select one of multiple food items included in the candidate list, the refrigerator (100) can perform a delivery process for the selected food item (or an unspecified package).

[0177] The refrigerator (100) can update the food DB according to the delivery process (S960). That is, the refrigerator (100) can delete or change information about food items that match the food items selected by the user among the food items stored in the food DB.

[0178] According to one embodiment, when a grocery item is received, the refrigerator (100) can sort the grocery delivery list within a preset time from the time of receipt to determine whether the received grocery item is a restock. At this time, the grocery delivery list may be a list generated based on a temporary storage grocery database. For example, the preset time may be 1 hour, but this is merely an example and may be changed according to user input. Alternatively, the preset time may be changed according to the user's usage pattern. For example, the refrigerator (100) may obtain pattern information regarding the time when the same grocery item is received after the grocery item is received, and set (or change) the preset time according to the pattern information. For example, if it is determined that the time pattern of the user receiving a grocery item and then receiving it again is 10 minutes, the refrigerator (100) may set the preset time to 10 minutes.

[0179] FIG. 10 is a diagram illustrating an embodiment of the present disclosure in which groceries are restocked within a preset time period. Furthermore, each of the first to third images illustrated in FIG. 10 may be comprised of multiple image frames, rather than a single image frame. However, for convenience of explanation, a single image frame will be assumed for the purpose of explanation.

[0180] The refrigerator (100) can identify through the first image (1010) that the milk was shipped at 15:10 on August 10, 2023.

[0181] And, the refrigerator (100) can acquire a second image (1020). At this time, the refrigerator (100) can identify whether the milk that was shipped within a preset time has been restocked. That is, the refrigerator (100) can identify that the milk has been shipped through the second image (1020), and can identify that the stocking time is within a preset time (e.g., 1 hour) from the shipping time, that is, 15:43 on August 10, 2023. Therefore, the refrigerator (100) can identify that the milk has been restocked through the second image (1020) and the stocking time. At this time, the refrigerator (100) can update the food DB according to the restocking of the milk. Specifically, the refrigerator (100) can delete the information about the restocked milk stored in the temporary storage food DB and re-store the information about the restocked milk in the food DB. In addition, the refrigerator (100) can update the food DB to maintain the time of receipt of the restocked milk as the initial receipt time rather than the restocked receipt time, and to add the restocking history (shipping time and restocking time, etc.).

[0182] In addition, the refrigerator (100) can acquire a third image (1030). At this time, the refrigerator (100) can identify whether milk shipped within a preset time has been restocked. At this time, the refrigerator (100) can identify that milk has been shipped through the third image (1030), and can identify that the stocking time is 17:13 on August 10, 2023, which is a preset time (e.g., 1 hour) after the shipping time. Therefore, the refrigerator (100) can identify that new milk has been shipped for the first time, not that milk has been restocked, through the second image (1020) and the stocking time. At this time, the refrigerator (100) can update the food DB according to the stocking of new milk. Specifically, the refrigerator (100) can store information on the new milk and information on the initial stocking time in the food DB.

[0183] In this way, the refrigerator (100) can manage the food DB by identifying the receipt or re-receipt of food, and can store the details of receipt or re-receipt in the food history DB.

[0184] FIG. 11 is a flowchart illustrating a method for managing the shipment of groceries according to one embodiment of the present disclosure.

[0185] The refrigerator (100) can identify the delivery of food (S1110). Specifically, the refrigerator (100) can determine the delivery of food detected from an image captured by the camera (110) and obtain characteristic information corresponding to the detected food.

[0186] The refrigerator (100) can identify whether a matching food item exists by comparing the shipped food item with the food item database (S1120). Specifically, the refrigerator (100) can identify whether, among the food items stored in the food item database, there exists a food item that has feature information that is similar to the shipped food item by a threshold value or higher (or a distance of less than a threshold value) with the feature information.

[0187] If it is identified that there is a food item matching the shipped food item (S1120-Y), the refrigerator (100) can determine whether the food item has been shipped (S1130). At this time, the refrigerator (100) can update the food item database (DB) according to the food item's shipment. In other words, the refrigerator (100) can delete or change information about the food item matching the shipped food item among the food items stored in the food item database.

[0188] If it is determined that there is no groceries matching the shipped groceries (S1120-N), the refrigerator (100) may provide a candidate list (S950). At this time, the candidate list is a list provided to the user to identify the shipped groceries, and may include at least one of n groceries corresponding to feature information most similar to the feature information corresponding to the shipped groceries, unspecified packaged groceries, and recently received groceries. At this time, the unspecified packaged groceries may be groceries received in an unspecified packaging state, such as a black plastic bag, a shopping bag, a box, etc., in which the contents cannot be confirmed.

[0189] The refrigerator (100) can receive a user input for selecting an unspecified packaged food item (S1160). That is, the refrigerator (100) can receive a user input for selecting (or touching) an unspecified packaged food item from among a plurality of food items included in a displayed candidate list.

[0190] The refrigerator (100) can match unspecified packaged food with shipped food (S1170). Specifically, the refrigerator (100) can match information about shipped food with information about unspecified packaged food.

[0191] The refrigerator (100) may output an inquiry message to the user (S1180). Specifically, the refrigerator (100) may output an inquiry message to the user asking whether the delivered food matches an unspecified packaged food. For example, the refrigerator (100) may output an inquiry message such as, "Are the apples being delivered in a black bag delivered at 3:30 PM on August 10, 2023?" In this case, the inquiry message may be provided through the display (123), but this is merely an example, and may be provided in an auditory form through the speaker (121). The refrigerator (100) may receive user input for a response to the inquiry message.

[0192] The refrigerator (100) can update the food DB (S1190). Specifically, if it is determined that the shipped food and the unspecified packaged food match (i.e., if a positive feedback input from the user in response to the inquiry message is received), the refrigerator (100) can delete or change the information about the unspecified packaged food stored in the food DB and update the information about the unspecified packaged food stored in the food history DB with the information about the shipped food. If it is determined that the shipped food and the unspecified packaged food do not match (i.e., if a negative feedback input from the user in response to the inquiry message is received), the refrigerator (100) can re-output the list of candidates for inquiry about the shipped food.

[0193] FIG. 12 is a diagram illustrating an embodiment of managing an unspecified packaging object according to one embodiment of the present disclosure. Incidentally, each of the first and second images described in FIG. 12 may be multiple image frames, rather than a single image frame. However, for convenience of explanation, a single image frame will be assumed for the purpose of explanation.

[0194] The refrigerator (100) can identify, through the first image (1210), that unspecified packaged food (e.g., black bag food) was received at 17:13 on August 10, 2023. The refrigerator (100) can store information about the unspecified packaged food in the food DB.

[0195] And, the refrigerator (100) can acquire a second image (1220). At this time, the refrigerator (100) can identify that the food being shipped is an apple through the second image (1220), and can identify that the apple was shipped at 17:15 on August 10, 2023. The refrigerator (100) can identify whether apples exist in the food DB. If it is identified that apples do not exist in the food DB, the refrigerator (100) can provide a candidate list. The refrigerator (100) can identify that apples correspond to unspecified packaged food based on a user input entered through the candidate list. The refrigerator (100) can update the unspecified packaged food stored in the food DB to apples, and can update the food DB so that unspecified packaged food stored in the food DB removes apples or reduces the number of apples. At this time, the refrigerator (100) may provide a query message to the user to confirm the number of apples stored in the unspecified packaged food. Alternatively, the refrigerator (100) may count the number of apples without deleting information about the unspecified packaged food from the food database. Furthermore, if unspecified packaging (e.g., a black bag) is identified as being shipped, the refrigerator (100) may delete information about the unspecified packaged food from the food database and store the counted number of apples in the food history database.

[0196] FIG. 13 is a flowchart illustrating a method for managing the delivery of groceries according to one embodiment of the present disclosure. Meanwhile, steps S1310 to S1330 illustrated in FIG. 13 correspond to steps S1110 to S1130 illustrated in FIG. 11 , and thus, any redundant description will be omitted.

[0197] The refrigerator (100) can identify whether any unspecified packaged food items are present and recognized prior to a preset time (S1340). The preset time may be one week, but is not limited thereto.

[0198] If it is identified that unspecified packaged food products are present and recognized as having been received prior to a preset time (S1340-Y), the refrigerator (100) can match the unspecified packaged food products with the shipped food products (S1350). Specifically, the refrigerator (100) can match information about the shipped food products with information about the unspecified packaged food products.

[0199] The refrigerator (100) can update the food DB (S1360). Specifically, the refrigerator (100) can update information about unspecified packaged food stored in the food DB and food history DB with information about shipped food.

[0200] The refrigerator (100) can process food items for shipment (S1370). That is, the refrigerator (100) can delete or change information about unspecified packaged food items stored in the food DB.

[0201] If it is determined that there are no unspecified packaged groceries that were received and recognized prior to the preset time (S1340-Y), the refrigerator (100) may update the groceries database (S1380). Specifically, the refrigerator (100) may update the groceries database with delivery information regarding the groceries delivered. Alternatively, the refrigerator (100) may provide a list of candidates for identifying the types of groceries delivered.

[0202] FIG. 14 is a flowchart illustrating a method for managing the receipt and delivery of bundled groceries according to one embodiment of the present disclosure.

[0203] The refrigerator (100) can detect the arrival of a bundled food item A (S1410). At this time, the bundled food item may be a food item that is stored in a bundled form by multiple individual food items. At this time, the second neural network model can be trained by assigning labels to the bundled food item during training, and the refrigerator (100) can identify whether it is a bundled food item using the second neural network model. That is, the refrigerator (100) can obtain information about the bundled food item A, such as information about whether it is a bundled food item, the type of the bundled food item, and the number of food items included in the bundled food item. In addition, the refrigerator (100) can obtain characteristic information about the bundled food item using the first neural network model.

[0204] The refrigerator (100) can update the food DB (S1420). That is, the refrigerator (100) can store information about bundled food A in the food DB. In particular, the refrigerator (100) can store information on the quantity of individual food items corresponding to bundled food A and information on the characteristics of bundled food A in the food DB.

[0205] The refrigerator (100) can detect the shipment of individual food items a (S1430). That is, the refrigerator (100) can detect the shipment of individual food items a constituting bundled food items A. In particular, the refrigerator (100) can obtain information on individual food items a, such as the type of individual food items and the number of individual food items shipped.

[0206] The refrigerator (100) can acquire characteristic information of individual food item A (S1440). At this time, the refrigerator (100) can acquire characteristic information of individual food item A using a first neural network model. The first neural network model can be trained so that the characteristic information of bundled food item A and individual food item A correspond.

[0207] In addition, the refrigerator (100) can match the bundled food A and the individual food a based on the acquired characteristic information (S1450). Specifically, the refrigerator (100) can identify that the bundled food A and the individual food a are of the same type of food based on the characteristic information of the bundled food A and the characteristic information of the individual food a, and can match the bundled food A and the individual food a based on the identification result.

[0208] The refrigerator (100) can update the food DB (S1460). At this time, the refrigerator (100) can update the food DB based on the number of individual food items a included in the bundled food items A and the number of shipped individual food items a. If the number of individual food items a included in the bundled food items A and the number of shipped individual food items a are the same, the refrigerator (100) can delete information about the bundled food items A stored in the food DB. If the number of shipped individual food items a is less than the number of individual food items a included in the bundled food items A, the refrigerator (100) can change the number of individual food items a included in the bundled food items A stored in the food DB.

[0209] FIG. 15 is a diagram illustrating an embodiment of managing the receipt and delivery of bundled food products according to one embodiment of the present disclosure. Furthermore, each of the first and second images described in FIG. 15 may be multiple image frames, rather than a single image frame. However, for convenience of explanation, a single image frame will be assumed for the purpose of explanation.

[0210] The refrigerator (100) can identify that bundled food A was delivered at 17:00 on March 24, 2023 through the first image (1510). The refrigerator (100) can obtain information about bundled food A through a second neural network model. At this time, the refrigerator (100) can store the type of bundled food (i.e., eggs) and the number of bundled food (e.g., 7) as information about bundled food A in the food DB. In addition, the refrigerator (100) can obtain feature information about bundled food A through the first neural network model.

[0211] And, the refrigerator (100) can obtain a second image (1220). At this time, the refrigerator (100) can identify the food item a being shipped through the second image (1220) and identify that eggs were shipped at 18:40 on March 24, 2023. The refrigerator (100) can obtain information on the individual food item a through the second neural network model. At this time, the refrigerator (100) can obtain the type of the individual food item (i.e., eggs) and the number of shipped individual food items (e.g., 1) as information on the individual food item a. In addition, the refrigerator (100) can obtain feature information on the individual food item a through the first neural network model. The refrigerator (100) can identify that the bundled food A and the individual food a are matching food items based on the characteristic information for the bundled food A and the characteristic information for the individual food a, and can change the number of bundled food items stored in the food DB based on the identification result. That is, the refrigerator (100) can change the number of bundled food items stored in the food DB from 7 to 6. Accordingly, the refrigerator (100) can provide more accurate information on food items within the refrigerator (100) by comprehensively managing the bundled food A and the individual food items a.

[0212] Figures 16 to 17b are flowcharts illustrating a method for managing the stocking of groceries based on the acquisition of specific information, according to one embodiment of the present disclosure. Incidentally, each of the first and second images illustrated in Figure 17b may be comprised of multiple image frames, rather than a single image frame. However, for convenience of explanation, a single image frame will be assumed for the purpose of explanation.

[0213] The refrigerator (100) can determine whether food has been received (S1610). That is, the refrigerator (100) can determine whether food has been received based on the direction of movement of the food object included in the image.

[0214] The refrigerator (100) can determine whether feature information about the food that has been received is obtainable (S1620). Specifically, if an area of ​​the food that has been received is covered by a user's hand by a certain size or more while the food is being received in the refrigerator, the refrigerator (100) cannot obtain feature information about the food from the image. In one embodiment, the refrigerator (100) can determine whether feature information about the food that has been received is obtainable based on the size of the food included in the image. In another embodiment, the refrigerator (100) can input an image (or an area corresponding to the food) into a first neural network model to obtain feature information about the food. In addition, the refrigerator (100) can determine whether food that corresponds to the feature information exists (i.e., whether similar feature information exists). If it is determined that food that corresponds to the feature information does not exist, the refrigerator (100) can determine that feature information about the food that has been received is not obtainable.

[0215] If it is identified that feature information about the food product that has been received is obtainable (S1620-Y), the refrigerator (100) can match the feature information about the image and the food product and store it in the food product DB (S1630), as described above.

[0216] If it is determined that feature information on the received food cannot be acquired (S1620-N), the refrigerator (100) can acquire information on an area corresponding to the food object (S1640). Specifically, the refrigerator (100) can acquire information on an area corresponding to the food object, such as texture information or information obtained by segmenting an area corresponding to the food object from an image. Specifically, as illustrated in FIG. 17A, when an image (1710) is acquired, the refrigerator (100) can segment an area (1720) corresponding to the food object to acquire information on the area corresponding to the food object.

[0217] The refrigerator (100) can match and store information on areas corresponding to images and food objects (S1650).

[0218] Afterwards, when food without a receipt history in the food DB is shipped, the refrigerator (100) can identify the shipped food based on the area corresponding to the food object.

[0219] Specifically, as illustrated in FIG. 17b, the refrigerator (100) acquires a first image (1710) and can identify, through the first image (1710), that the food was delivered at 5:13 PM on August 10, 2023. However, the refrigerator (100) can identify that the food is obscured by the user's hand, making it impossible to acquire characteristic information about the food. The refrigerator (100) can acquire and store information about an area corresponding to the food.

[0220] And, the refrigerator (100) can obtain a second image (1720). At this time, the refrigerator (100) can identify that the food being shipped is milk through the second image (1720), and that the milk was shipped at 17:15 on August 10, 2023. In one embodiment, the refrigerator (100) can identify that the shipped milk is a food for which feature information cannot be obtained previously based on information (texture information or segmented image information) on an area corresponding to the food. The refrigerator (100) can update the food DB by matching the food for which feature information cannot be obtained with the shipped food (milk). In another embodiment, when milk, which is a food for which there is no history of stocking in the food DB, is shipped, the refrigerator (100) can generate a candidate list as described above. At this time, the candidate list may include n groceries corresponding to feature information most similar to the feature information corresponding to the shipped groceries, unspecified packaged groceries, and recently received groceries, as well as groceries for which feature information has not been acquired. When groceries for which feature information has not been acquired (i.e., an area corresponding to a groceries object) are selected through the candidate list, the refrigerator (100) may update the groceries DB by matching the groceries for which feature information has not been acquired with the shipped groceries. In another embodiment, when milk, which is a groceries for which there is no history of receipt in the groceries DB, is shipped, the refrigerator (100) may ask the user a question through the speaker (121). For example, the refrigerator (100) may output a question message such as "Is the food that was put in the refrigerator at 17:13 on August 10, 2023 milk?", and may update the groceries DB by matching the groceries for which feature information has not been acquired with the shipped groceries through the user's feedback (e.g., "Yes"). At this time, the refrigerator (100) can output a question message to the user based on the arrival time of the food for which characteristic information has not been obtained and the shipment time of the shipped food.

[0221] Additionally, the refrigerator (100) can train a first neural network model based on an image including an area corresponding to a food object and the food that has been shipped. That is, when the area corresponding to the food object is input into the first neural network model, the refrigerator (100) can train the first neural network model to acquire feature information corresponding to the food that has been shipped.

[0222] FIG. 18 is a flowchart illustrating a method for managing the receipt of groceries according to identification information corresponding to a hand object, according to one embodiment of the present disclosure.

[0223] Food storage may occur (S1810). At this time, the refrigerator (100) may capture an image of the food storage.

[0224] The refrigerator (100) can detect hand objects and food objects from the captured images (S1820).

[0225] The refrigerator (100) can separate an area corresponding to the detected hand object (S1830). That is, the refrigerator (100) can separate an area corresponding to the hand object among the hand object and food objects detected from the captured image.

[0226] The refrigerator (100) can obtain identification information corresponding to a hand object (S1840). Specifically, the refrigerator (100) can obtain identification information corresponding to the hand object by identifying the shape of the fingers, nails, back of the hand, or palm included in the hand object. At this time, the identification information corresponding to the hand object may be pre-stored. That is, the refrigerator (100) can pre-store identification information corresponding to the hand object by taking a picture of the hand of each user and registering it.

[0227] The refrigerator (100) can acquire feature information corresponding to a food object (S1850). Specifically, the refrigerator (100) can acquire feature information corresponding to a food object by inputting an image into a first neural network model.

[0228] The refrigerator (100) can store identification information corresponding to a hand object, an image, and feature information corresponding to a food object in a food DB (S1860). That is, the refrigerator (100) can store not only feature information corresponding to an image and a food object, but also information about the user who has received the food.

[0229] FIG. 19 is a flowchart illustrating a method for managing the delivery of groceries according to identification information corresponding to a hand object, according to one embodiment of the present disclosure.

[0230] Food delivery may occur (S1910). At this time, the refrigerator (100) may capture an image of the food delivery.

[0231] The refrigerator (100) can detect hand objects and food objects from the captured images (S1920).

[0232] The refrigerator (100) can separate an area corresponding to the detected hand object (S1930). That is, the refrigerator (100) can separate an area corresponding to the hand object among the hand object and food objects detected from the captured image.

[0233] The refrigerator (100) can obtain identification information corresponding to a hand object (S1940). Specifically, the refrigerator (100) can obtain identification information corresponding to a hand object by identifying the shape of the fingers, nails, back of the hand, or palm included in the hand object.

[0234] The refrigerator (100) can determine whether the identification information corresponding to the hand object upon delivery matches the identification information corresponding to the hand object upon receipt (S1950). That is, the refrigerator (100) can determine whether the identification information corresponding to the hand object upon delivery of the food product is identical to the identification information corresponding to the hand object upon receipt.

[0235] If it is determined that the identification information corresponding to the hand object at the time of shipment matches the identification information corresponding to the hand object at the time of receipt (S1950-Y), the refrigerator (100) can store a single-user entry / exit pattern (S1960). In this case, the single-user entry / exit pattern may refer to a pattern in which a single user enters and exits groceries.

[0236] In addition, the refrigerator (100) can update the food DB (S1990). That is, the refrigerator (100) can remove the food items shipped from the food DB and store identification information corresponding to the hand objects related to the receipt and shipment in the food history DB.

[0237] If it is determined that the identification information corresponding to the hand object at the time of shipment does not match the identification information corresponding to the hand object at the time of receipt (S1950-Y), the refrigerator (100) may provide an alarm message notifying of another user's shipment (S1970). Specifically, if the refrigerator (100) is shipped by another user, the refrigerator (100) may notify of the other user's shipment through the display (123) or the user terminal.

[0238] The refrigerator (100) can store a multi-user entry / exit pattern (S1980). In this case, the multi-user entry / exit pattern may refer to a pattern in which multiple users enter and exit groceries.

[0239] In addition, the refrigerator (100) can update the food DB (S1990). That is, the refrigerator (100) can remove the food items shipped from the food DB and store identification information corresponding to the hand objects related to the receipt and shipment in the food history DB.

[0240] FIG. 20 is a diagram illustrating an embodiment of managing the receipt and delivery of groceries based on identification information corresponding to a hand object, according to one embodiment of the present disclosure. Incidentally, each of the first to fifth images described in FIG. 20 may be multiple image frames, rather than a single image frame. However, for convenience of explanation, a single image frame will be assumed for the purpose of explanation.

[0241] The refrigerator (100) can identify the receipt and delivery of food based on the first to fifth images (2010 to 2050) as illustrated in FIG. 20. Specifically, the refrigerator (100) can identify that "milk was delivered by user A at 15:10 on August 10, 2023" through the first image (2010). The refrigerator (100) can identify that "snacks were delivered by user B at 15:21 on August 10, 2023" through the second image (2020). The refrigerator (100) can identify that "milk was delivered by user B at 15:33 on August 10, 2023" through the third image (2030). The refrigerator (100) can identify that "the juice was shipped by user C at 16:10 on August 10, 2023" through the fourth image (2040). The refrigerator (100) can identify that "the juice was shipped by user C at 16:13 on August 10, 2023" through the fifth image (2050).

[0242] As described above, the refrigerator (100) can identify and manage information about the type of groceries being put in and out through the hand object and the user putting in and out the groceries.

[0243] For example, if the same food is delivered and then restocked by the same user within a preset time (e.g., 1 hour), the refrigerator (100) can identify that the food has been restocked. However, if the same food is delivered and then restocked by a different user within a preset time (e.g., 1 hour), the refrigerator (100) can identify that new food has been delivered.

[0244] For example, the refrigerator (100) may provide an alarm to the user if the user who received the groceries is different from the user who returned the same groceries. In this case, the refrigerator (100) may provide the alarm via the display (123) or a user terminal.

[0245] For example, the refrigerator (100) can extract and store food pattern information. This food pattern information may include information and pattern information regarding food items received or shipped during a specific time period. The refrigerator (100) can utilize the food pattern information to update its grocery list. Furthermore, the refrigerator (100) can use the food pattern information to provide user pattern information to users and, based on the user pattern information, provide personalized recommendation services to each user.

[0246] For example, the refrigerator (100) may provide a sharing function among users using the refrigerator (100). For example, when User A places milk in the refrigerator, the refrigerator (100) may share the information via a messaging application or on the display (123) on the refrigerator. In addition, the refrigerator (100) may track the user who places the milk placed by User A and the time of shipment.

[0247] FIG. 21 is a diagram for explaining an object recognition model according to one embodiment of the present disclosure.

[0248] The object recognition model (2120) is a model for recognizing the type of a food object and may be referred to as a second neural network model. In particular, the object recognition model (2120) may include a plurality of layers to recognize the type of a food object. When a captured image (2110) is input to the object recognition model (2120), the object recognition model (2120) may output the type of the food object included in the captured image (2110) and its probability value. For example, the object recognition model (2120) may output the output value "Tomato 85%" (2130). However, this is only one embodiment, and the object recognition model (2120) may output a probability value for each of a plurality of candidate objects, and the refrigerator (100) may identify the candidate object having the highest probability value among the output probability values ​​as the type of the food object. Additionally, the object recognition model (2120) can output information on the type of food objects by classifying it into multiple levels. For example, the object recognition model (212) can output “Major category (fresh food) - Subcategory (vegetables) - Detailed food name (asparagus)” or “Major category (processed food) - Subcategory (canned food) - Detailed food name (canned tuna)” as output values.

[0249] According to one embodiment of the present disclosure, the object recognition model (2120) can be implemented with various neural network models such as a convolutional neural network (CNN), a transformer, etc.

[0250] FIG. 22 is a flowchart illustrating a method for managing the receipt of groceries according to whether the groceries are normally received, according to one embodiment of the present disclosure.

[0251] The refrigerator (100) can recognize the receipt of food (S2210). In this case, the receipt of food may mean recognizing the movement of a food object moving toward the inside of the refrigerator.

[0252] The refrigerator (100) can identify whether food has been normally received (S2220). For example, if a captured image is input into the second neural network model, and the refrigerator (100) cannot identify the type of food object included in the captured image (or, if the probability that the food object corresponds to a specific type is below a preset value (e.g., 50%)), it can identify the food as abnormally received.

[0253] If the groceries are identified as being normally received (S2220-Y), the refrigerator (100) can add the groceries to the grocery list and output a success message (S2230). That is, the refrigerator (100) can add the received groceries to the grocery list and output a success message indicating that the groceries have been received. At this time, the success message may include information about the received groceries and information about the groceries being received.

[0254] If abnormal food delivery is identified (S2220-N), the refrigerator (100) may output a failure message (S2240). That is, the refrigerator (100) may output a failure message indicating a failure in food delivery recognition. The failure message may include a message requesting information about the delivered food.

[0255] FIG. 23 is a flowchart illustrating a method for managing the delivery of groceries depending on whether the groceries are normally delivered, according to one embodiment of the present disclosure.

[0256] The refrigerator (100) can recognize the delivery of food (S2310). In this case, the delivery of food may mean recognizing the movement of a food object moving outward from the refrigerator.

[0257] The refrigerator (100) can identify whether food has been delivered normally (S2320). For example, if a captured image is input into the second neural network model, the refrigerator (100) can identify the food as being delivered abnormally if it cannot identify the type of food object included in the image.

[0258] If an abnormal food delivery is identified (S2320-N), the refrigerator (100) may output a failure message (S2330). That is, the refrigerator (100) may determine that the food delivery is abnormal and output a failure message. In this case, the abnormal food delivery may be an event in which the type of the delivered food is not recognized. In this case, the failure message may include a message requesting information about the delivered food.

[0259] If the groceries are identified as being normally shipped (S2320-Y), the refrigerator (100) can identify whether groceries matching the recognized shipped groceries exist in the groceries DB (S2340).

[0260] If it is identified that a food item matching the recognized delivery item exists in the food DB (S2340-Y), the refrigerator (100) can update the food DB and output a success message (S2350). That is, the refrigerator (100) can remove or change information about the delivered food item in the food DB and output a success message indicating that the delivery item has been recognized. At this time, the success message may include information about the delivered food item and delivery information.

[0261] If it is determined that the food matching the recognized food item for shipment does not exist in the food DB (S2340-N), the refrigerator (100) may output a failure message indicating a failure in the normal food shipment recognition (S2360). At this time, the failure message may include a message requesting information about the shipped food item.

[0262] FIGS. 24a and 24b are diagrams illustrating a UI that provides information on groceries being received or shipped, according to one embodiment of the present disclosure.

[0263] The refrigerator (100) can update the food database by reflecting information about the food items received and shipped when the food items are received and shipped. Furthermore, the refrigerator (100) can provide a food list and information about the food items received and shipped based on the updated food database. At least one of the food list and the information about the food items received and shipped may be output each time the door is opened and closed. However, this is merely an example, and may be output based on user input (e.g., a user touch).

[0264] For example, the refrigerator (100) may provide a pop-up screen (2410) including information on incoming and outgoing food, as illustrated in FIG. 24A. At this time, the information on incoming and outgoing food may be output on the display (123) of the refrigerator (100), but this is merely an example, and may be output on the display of a user terminal. Meanwhile, the refrigerator (100) providing information on incoming and outgoing food through the pop-up screen (2410) is merely an example, and may be provided in various forms (e.g., images, lists, etc.).

[0265] Additionally, the refrigerator (100) can provide a voice assistant alarm through the speaker (121). Alternatively, the refrigerator (100) can be connected to an external device (e.g., an AI speaker, etc.) to provide information on food items that have been brought in and out. For example, the refrigerator (100) can output a voice message, such as "Cabbage, cheese, milk, and raw chicken have arrived on the refrigerator shelf. 2L of milk, eggs, wine, and oranges have left" through the speaker (121) or the speaker of the external device.

[0266] Additionally, the refrigerator (100) may provide information for outputting a notification message (or push message) to a user terminal. In this case, the refrigerator (100) may provide information for outputting a notification message to a plurality of registered user terminals.

[0267] Additionally, the refrigerator (100) can provide a notification message in conjunction with a display device (e.g., TV, etc.) among devices connected in the home.

[0268] In addition, the refrigerator (100) can obtain recipe information based on information about the delivered food. For example, the refrigerator (100) can obtain recipe information related to the delivered food from among the recipe information stored in the memory (160), and can obtain recipe information related to the delivered food from an external server. In addition, the refrigerator (100) can transmit the obtained recipe information to a device (e.g., an oven, a microwave oven, a hood) related to cooking the delivered food among devices connected in the home or to a user terminal. Accordingly, the user can more conveniently check the recipe information related to the delivered food. In addition, the refrigerator (100) can identify the storage location of the delivered food and provide a UI screen (2420) including an internal image of the inside of the refrigerator and the storage location of the delivered food, as illustrated in FIG. 24b.

[0269] In addition, the refrigerator (100) can provide the user with information such as whether the camera is operating and whether the food has been entered or exited through sound effects. At this time, the sound effects may be output through the speaker (121) of the refrigerator (100), but this is only one embodiment, and may be output through the speaker of an external device connected to the refrigerator (100). Through this, even if the entry / exit details are not constantly displayed on the display screen, the user can be immediately informed through sound effects of whether the camera (110) is operating normally, whether the food has been entered or exited normally, or whether there has been an error or failure to recognize the information.

[0270] In addition, the refrigerator (100) may provide a sound effect for normal operation of the camera as a first sound effect, a sound effect for normal recognition of food receipt as a second sound effect, a sound effect for normal recognition of food delivery as a third sound effect, a sound effect for a food receipt error (non-recognition or misrecognition of food) as a fourth sound effect, and a sound effect for a food delivery error (non-recognition or misrecognition of food) as a fifth sound effect. At this time, the first to fifth sound effects may be different from each other, but this is only an example, and the second and third sound effects may be the same, and the fourth and fifth sound effects may also be the same. The first sound effect may be output when the camera starts operating normally after the door is opened. The second or third sound effect may be output whenever one food item is normally received or delivered. The fourth or fifth sound effect may be output whenever the user's hand / food is detected, but the type of food item is not identified.

[0271] FIGS. 25A to 25C are drawings illustrating a candidate list for shipped food products according to one embodiment of the present disclosure.

[0272] When the availability of food is determined, the refrigerator (100) can not only match the most similar food stored in the food database, but also output a candidate list including food items with similar characteristic information. The number of items included in the candidate list can be changed, and a user-specified number can be output.

[0273] In particular, when the food storage or delivery management process is completed, the refrigerator (100) can provide the user with information on the food items received or delivered in a list format. Furthermore, when one of the food items included in the list is selected, the refrigerator (100) can provide a list of candidates for replacing the selected food item with another food item.

[0274] For example, the refrigerator (100) can output a list (2510) including information on incoming and outgoing groceries as illustrated in FIG. 25a. At this time, if a first groceries (2511) is selected from among the groceries included in the list (2510), the refrigerator (100) can display a candidate list (2520) including n candidate groceries (2521, 2522, 2523) having similar feature information to that of the first groceries (2511), as illustrated in FIG. 25b. At this time, the selected first groceries (2511) can be displayed to be distinguished from other groceries. In addition, as illustrated in FIG. 25b, if a second candidate groceries (2522) is selected from among the candidate list, the refrigerator (100) can change the first groceries (2511) to the second candidate groceries (2522) and display them as illustrated in FIG. 25c. Additionally, a new fourth candidate food item (2524) can be added to the candidate list. This allows the user to change the type of the recognized food item to a different type, thereby increasing user convenience.

[0275] FIGS. 26A and 26B are drawings for explaining an embodiment in which a user corrects a misrecognition of a shipped food product according to one embodiment of the present disclosure.

[0276] The refrigerator (100) can display a pop-up screen with a list of food items shipped from the food database at set intervals. Specifically, the refrigerator (100) can store the list of shipped food items in the food history database and then display it at set intervals. The set interval may be, for example, one week, but this is merely an example and can be changed by user input.

[0277] For example, the refrigerator (100) may display a pop-up screen (2610) containing information about the processed food products, as illustrated in FIG. 26a.

[0278] When the pop-up screen (2610) is displayed, the user can check the details of the processed food items, and if there are foods that have been processed for shipment but have not actually been shipped, they can be added back to the food DB. For example, when the first food item (e.g., eggs) and the second food item (e.g., carrots) are selected on the pop-up screen (2610) and the "Add to Food List" button is selected, the refrigerator (100) can add the first food item and the second food item to the food DB again and display a guidance message (2620) saying "Added to Food List" as illustrated in FIG. 26b.

[0279] For example, the refrigerator (100) may display a button for purchasing processed food. When the button is selected, the refrigerator (100) may link to a website for purchasing processed food and display a food purchase page. This allows the user to more easily purchase processed food.

[0280]

[0281] Meanwhile, the order of the flowcharts of the various embodiments described above is only one example, and the order of each step of the flowchart may be changed, and of course, the order of the flowcharts may be performed simultaneously.

[0282] Meanwhile, according to one embodiment of the present disclosure, the processor (170) controls input data to be processed according to predefined operation rules or artificial intelligence models stored in the memory (160). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0283] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0284] An artificial intelligence model (e.g., first and second object detection networks) may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0285] A learning algorithm is a method for training a target device using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0286] Meanwhile, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0287] The methods according to various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., a refrigerator) according to the disclosed embodiments.

[0288] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0289] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0290] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the refrigerator, A body including a storage room; A door rotatably coupled to the main body to open and close the storage room, the door including a door bin; A camera located in the main body and photographing the inside of the main body and the inside of the door; memory for storing at least one instruction; and When a trigger signal is detected, an image is acquired by photographing at least a portion of the interior of the main body and the door through the camera, Detecting a first food object included in the acquired image, Tracking the first grocery object to identify whether the first grocery object is being delivered to or shipped from the refrigerator; When the first grocery object is identified as being received, the image is input into the learned first neural network model to obtain first feature information corresponding to the first grocery object, A refrigerator including a processor that matches the image and the acquired first feature information and stores them in a food database.

2. In paragraph 1, The above processor, Inputting the above image into the learned second neural network model to obtain information about the first food object, Store information about the food product together with the image and the first feature information obtained in the food product database, Information about the above first grocery object, A refrigerator comprising at least one of a type of the first food object, a product name of the first food object, a manufacturer of the first food object, and a capacity of the first food object.

3. In paragraph 1, The above processor, Crop the area containing the first grocery object from the above image, A refrigerator that inputs an area including the first food object into the learned first neural network model to obtain first feature information corresponding to the first food object.

4. In paragraph 1, By inputting the above image into the learned first neural network model, second feature information corresponding to the first food product object is obtained, The above processor, When the first grocery object is identified as being shipped, it is identified whether there is a grocery item that matches the second characteristic information among the grocery items stored in the grocery database. A refrigerator that deletes the groceries matching the second feature information from the groceries database when it is identified that there is a groceries matching the second feature information.

5. In paragraph 4, The above processor, After identifying the first grocery object as being shipped, identify whether the second grocery object has been received within a preset time, If it is identified that the second grocery object has been received within the above preset time, it is identified whether the first grocery object and the second grocery object match, A refrigerator that identifies a restocking event of the first grocery object when the first grocery object and the second grocery object are identified as matching, and stores information related to the restocking event in the grocery database.

6. In paragraph 5, The above processor, If it is identified that the second grocery object is not received within the above-mentioned preset time or that the first grocery object and the second grocery object do not match, it is identified whether there is a grocery object corresponding to the first grocery object among the grocery objects stored in the grocery database, A refrigerator that deletes the groceries corresponding to the first groceries object from the groceries database when it is identified that there is a groceries corresponding to the first groceries object.

7. In paragraph 6, The above processor, If it is identified that there is no food corresponding to the first food object, a candidate list for the first food object is provided. The above list of candidates is: A refrigerator including n grocery objects, an unspecified packaging object, and at least one of a recently received grocery object, each of which has feature information most similar to third feature information corresponding to the first grocery object.

8. In paragraph 6, The above processor, If it is identified that there is no food corresponding to the first food object above, it is identified whether there is an unspecified package object that was delivered before the preset time, A refrigerator that updates the grocery database by determining that the first grocery object and the unspecified grocery object correspond when the unspecified grocery object is identified as existing.

9. In paragraph 1, The above processor, When the first grocery object is identified as being received, it is identified whether acquisition of first characteristic information corresponding to the first grocery object is possible, If it is determined that acquisition of the first feature information corresponding to the first food object is impossible, information on the area corresponding to the first food object is acquired from the image, A refrigerator that matches information about an area corresponding to the image and the first grocery object and stores the information in the grocery database.

10. In paragraph 1, The above processor, Detect a hand object from the above image, Obtain identification information for the detected hand object based on multiple stored hand objects, A refrigerator that matches identification information for the hand object, the image, and the first feature information and stores them in the food database.

11. A method for controlling a refrigerator, comprising: a main body including a storage compartment; a door including a door bin, the door being rotatably coupled to the main body to open and close the storage compartment; and a camera positioned on the main body and photographing the inside of the main body and the inside of the door. When a trigger signal is detected, a step of capturing an image by photographing at least a part of the inside of the main body and the door through the camera; A step of detecting a first food object included in the acquired image; A step of tracking the first grocery object to identify whether the first grocery object is being delivered to or shipped from the refrigerator; When the first grocery object is identified as being received, a step of inputting the image into a learned first neural network model to obtain first feature information corresponding to the first grocery object; A control method comprising: a step of matching the image and the acquired first feature information and storing them in a grocery database.

12. In paragraph 11, The above control method is, Further comprising a step of inputting the image into a learned second neural network model to obtain information about the first food product object; The above saving steps are: Store information about the food product together with the image and the first feature information obtained in the food product database, Information about the above first grocery object, A control method comprising at least one of the type of the first food object, the product name of the first food object, the manufacturer of the first food object, and the capacity of the first food object.

13. In paragraph 11, The above control method is, A step of cropping an area including the first grocery object from the image; The step of obtaining the above first characteristic information is: A control method for inputting an area including the first food object into the learned first neural network model to obtain first feature information corresponding to the first food object.

14. In paragraph 11, The above control method is, If the first grocery object is identified as being shipped, a step of inputting the image into the learned first neural network model to obtain second feature information corresponding to the first grocery object; A step of identifying whether there is a food product that matches the second characteristic information among the food products stored in the food product database; and A control method comprising: a step of deleting a food product matching the second characteristic information from the food product database if it is identified that a food product matching the second characteristic information exists.

15. In paragraph 14, The above control method is, A step of identifying whether a second grocery object has been received within a preset time after the first grocery object has been identified as being shipped; If it is identified that the second grocery object has been received within the above preset time, a step of identifying whether the first grocery object and the second grocery object match; and A control method comprising: a step of identifying a restocking event of the first grocery object and storing information related to the restocking event in the grocery database when the first grocery object and the second grocery object are identified as matching.

Images