Refrigerator and method for identifying objects in refrigerator

Abstract

Disclosed is a method for operating a refrigerator, the method comprising the steps of: tracking a position of a user's hand entering a refrigerator by using at least one camera; comparing the tracked position of the user's hand with position information of objects stored in a memory; identifying at least one object selected by the user from among objects stored in the refrigerator, on the basis of a result of the comparison; and providing information related to the identified at least one object.

Description

Refrigerator and refrigerator object identification method

[0001] The present disclosure relates to a refrigerator and a method of operating the refrigerator for identifying an object selected by a user.

[0002] A refrigerator is a device designed to preserve food for a long time without spoiling, utilizing freezing or refrigeration methods. A refrigerator's fundamental function is to maintain the internal temperature at a set level by creating a cooling environment using refrigerant.

[0003] Recently, as more and more food is stored in the refrigerator storage space, technologies that enable users to easily manage stored food are in demand.

[0004] According to one embodiment of the present disclosure, a refrigerator may include at least one camera; a memory storing storage location information of objects placed in the refrigerator; and at least one processor. The at least one processor may track the location of a user's hand placed in the refrigerator using the at least one camera. The at least one processor may compare the location of the user's hand with the storage location information of objects stored in the memory. Based on the result of comparing the location of the user's hand with the storage location information of objects stored in the memory, the at least one processor may identify at least one object selected by the user among the objects placed in the refrigerator. The at least one processor may provide information related to the at least one object selected by the user.

[0005] A method for identifying an object in a refrigerator according to one embodiment of the present disclosure may include: tracking a position of a user's hand placed in the refrigerator using at least one camera; comparing the tracked position of the user's hand with storage location information of objects stored in a memory; identifying at least one object selected by the user among objects placed in the refrigerator based on a result of the comparison; and providing information related to at least one object selected by the user.

[0006] A method for identifying an object in a refrigerator according to one embodiment of the present disclosure may include: at least one processor, a step of tracking a user's hand placed in the refrigerator using at least one camera; a step of analyzing an image frame acquired at the time when a specific gesture is detected from the user's hand to identify at least one object selected by the user among objects placed in the refrigerator; and a step of providing information related to at least one object selected by the user.

[0007] FIG. 1 is a drawing for explaining a refrigerator according to one embodiment of the present disclosure.

[0008] FIG. 2 is a drawing for explaining at least one camera according to one embodiment of the present disclosure.

[0009] FIG. 3 is a drawing for explaining a type of camera according to one embodiment of the present disclosure.

[0010] FIG. 4 is a flowchart illustrating a method for a refrigerator according to one embodiment of the present disclosure to determine whether to receive or ship an object.

[0011] FIG. 5 is a drawing for explaining an operation of a refrigerator tracking a user's hand or an object according to one embodiment of the present disclosure.

[0012] FIG. 6 is a diagram for explaining an object management table according to one embodiment of the present disclosure.

[0013] FIG. 7 is a flowchart for explaining an object identification method of a refrigerator according to one embodiment of the present disclosure.

[0014] FIG. 8 is a drawing for explaining an operation of a refrigerator providing information related to an object through voice according to one embodiment of the present disclosure.

[0015] FIG. 9A is a diagram illustrating an operation of a refrigerator providing a list of selected products according to one embodiment of the present disclosure.

[0016] FIG. 9b is a diagram for explaining editing of a selection product list according to one embodiment of the present disclosure.

[0017] FIG. 10 is a flowchart illustrating a method for a refrigerator to provide recipe information according to one embodiment of the present disclosure.

[0018] FIG. 11 is a diagram illustrating an operation of a refrigerator according to one embodiment of the present disclosure to provide additional information including recipe information.

[0019] FIG. 12 is a flowchart illustrating a method for a refrigerator to provide purchase information according to one embodiment of the present disclosure.

[0020] FIG. 13 is a drawing for explaining an operation of a refrigerator providing purchase information according to one embodiment of the present disclosure.

[0021] FIG. 14 is a flowchart illustrating a method for a refrigerator according to an embodiment of the present disclosure to perform a control command according to a user's gesture.

[0022] FIG. 15 is a drawing for explaining an operation of a refrigerator according to an embodiment of the present disclosure to perform a control command according to a user's gesture.

[0023] FIG. 16 is a flowchart illustrating a method for a refrigerator to output a notification message as a voice according to one embodiment of the present disclosure.

[0024] FIG. 17 is a drawing for explaining an operation of a refrigerator according to one embodiment of the present disclosure to output a notification message as voice.

[0025] FIG. 18 is a flowchart illustrating a method for a refrigerator according to one embodiment of the present disclosure to identify at least one object selected by a user.

[0026] FIG. 19a is a block diagram illustrating the function of a refrigerator according to one embodiment of the present disclosure.

[0027] FIG. 19b is a block diagram illustrating the function of a refrigerator according to one embodiment of the present disclosure.

[0028] FIG. 20 is a drawing for explaining a communication system of a refrigerator according to one embodiment of the present disclosure.

[0029] FIG. 21 is a drawing for explaining an operation of a refrigerator interworking with a user terminal according to one embodiment of the present disclosure.

[0030] FIG. 22 is a diagram for explaining an operation of providing a list of selected products through a user terminal according to one embodiment of the present disclosure.

[0031] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0032] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0033] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0034] Throughout this disclosure, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components may be included, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used in this disclosure refer to a unit that processes at least one function or operation, and "part" and "module" may be implemented as hardware or software, or as a combination of hardware and software.

[0035] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be stored in separate portions across multiple different memories.

[0036] Unless the context clearly dictates otherwise, singular forms (e.g., "a," "an," and "the") are to be understood to include plural referents. Thus, for example, the description "a component surface" may also include reference to one or more of such surfaces.

[0037] All functions or operations described in this document may be performed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing, and may include circuitry such as an Application Processor (AP), a Communication Processor (CP), a Graphical Processing Unit (GPU), a Neural Processing Unit (NPU), a Microprocessor Unit (MPU), a System on Chip (SoC), or an Integrated Chip (IC).

[0038] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0039] A refrigerator according to one embodiment may include a body.

[0040] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0041] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0042] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0043] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0044] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0045] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0046] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0047] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0048] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0049] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0050] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0051] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door basket for storing items.

[0052] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0053] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0054] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0055] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0056] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device (expander), and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment through heat generation and cooling through the Peltier effect.

[0057] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0058] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0059] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0060] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0061] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0062] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0063] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0064] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0065] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0066] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0067] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0068] The communication module (communication interface) can communicate with external devices such as servers, mobile devices, and other home appliances via a nearby access point (AP). The access point (AP) can connect the local area network (LAN) where the refrigerator or user device is connected to the wide area network (WAN) where the server is connected. The refrigerator or user device can then connect to the server via the wide area network (WAN).

[0069] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0070] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0071] Hereinafter, refrigerators according to various embodiments will be specifically described with reference to the attached drawings.

[0072] FIG. 1 is a drawing for explaining a refrigerator (1000) according to one embodiment of the present disclosure.

[0073] According to one embodiment of the present disclosure, a refrigerator (1000) may be an electronic device (or home appliance) that refrigerates or freezes food. The refrigerator (1000) may also store medicine, alcoholic liquor, or cosmetics in addition to food.

[0074] Referring to FIG. 1, a refrigerator (1000) according to one embodiment of the present disclosure may include at least one camera. For example, the refrigerator (1000) may include a camera (1401) (hereinafter, also referred to as a main camera) at the top center. The refrigerator (1000) may monitor objects entering or leaving the refrigerator (1000) through the camera (1401). In the present disclosure, an object is an element that can be stored in a refrigerator (1000), and may include, but is not limited to, for example, food (e.g., fresh food such as fruits and vegetables, cooked food, processed products, seasonings, retort food), containers for storing side dishes or food (e.g., translucent containers, transparent containers, opaque containers), beverages (e.g., mineral water, carbonated water, yogurt, coffee, milk, etc.), canned goods, sauces (e.g., ketchup, mayonnaise, salad dressing, steak sauce, etc.), medicines, alcohol, cosmetics, etc. The operation of recognizing an object that is received or shipped by the refrigerator (1000) will be described in detail later with reference to FIG. 4.

[0075] According to one embodiment of the present disclosure, the refrigerator (1000) can track the position of a user's hand using a camera (1401) and recognize an object selected by the user. For example, when the door (10) is open (101), the refrigerator (1000) can activate the camera (1401) to identify an object selected by the user among stored objects. According to one embodiment of the present disclosure, the refrigerator (1000) can add the object selected by the user to a selection product list (100). The selection product list (100) can be stored and managed in a storage standby database (e.g., a temporary storage DB). The storage standby database (e.g., a temporary storage DB) can be a DB that stores history only for a certain period of time (e.g., 10 minutes) after the user selects an object. The operation of the refrigerator (1000) to identify an object selected by the user will be described in detail later with reference to FIG. 7.

[0076] According to one embodiment of the present disclosure, when the door (10) is in an open state (101), it is difficult for the user to check the information output on the display unit (1300). Therefore, when there is urgent or useful information related to an object selected by the user, the refrigerator (1000) may provide information related to the object selected by the user through a speaker at the time the object selected by the user is recognized. The operation of the refrigerator (1000) providing information related to the object selected by the user through a voice will be described in detail later with reference to FIG. 8.

[0077] According to one embodiment of the present disclosure, the refrigerator (1000) can provide information related to an object selected by a user on the display unit (1300) when the door (10) is switched from an open state to a closed state (102). For example, the refrigerator (1000) can output a list of selected products stored in a storage standby database (e.g., temporary storage DB) on the display unit (1300). The list of selected products output on the display unit (1300) can be implemented in the form of a table including identification information of the objects (e.g., product name), or can be implemented in the form of an image inside the refrigerator including an image of the selected object.

[0078] According to one embodiment of the present disclosure, the refrigerator (1000) may output, through the display unit (1300), information related to an object selected by a user, such as an image of the object and the storage location of the object. In addition, the refrigerator (1000) may obtain additional information related to the object selected by the user from a memory or an external server, and output the additional information through the display unit (1300). For example, the refrigerator (1000) may further output, on the display unit (1300), at least one of the object's expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, or appropriate eating period information. In addition, according to one embodiment of the present disclosure, the refrigerator (1000) may provide purchase information of the object selected by the user. According to one embodiment of the present disclosure, when the user selects multiple objects, the refrigerator (1000) may provide recipe information corresponding to a combination of the multiple objects selected by the user. The operation of the refrigerator (1000) to provide information related to an object selected by the user will be described in detail later with reference to FIGS. 9a to 13.

[0079] Therefore, according to one embodiment of the present disclosure, the refrigerator (1000) can accurately recognize an object selected by a user by tracking the position of a hand, and quickly transmit information related to the object selected by the user to the user, thereby increasing the efficiency of the user's management of objects inside the refrigerator (1000).

[0080] Hereinafter, with reference to FIGS. 2 and 3, we will look a little more closely at at least one camera of the refrigerator (1000) for recognizing an object being brought in or out or a user's hand.

[0081] FIG. 2 is a drawing for explaining at least one camera according to one embodiment of the present disclosure.

[0082] According to one embodiment of the present disclosure, the refrigerator (1000) may include one camera or multiple cameras. For example, the refrigerator (1000) may include only one main camera (1401) located at the top center (e.g., in front of the top table). Additionally, the refrigerator (1000) may include sub-cameras (1402) in addition to the main camera (1401).

[0083] Referring to 210 of FIG. 2, the sub-cameras (1402) can be positioned at the top of the storage room (or door bin) for each storage space. When the sub-cameras (1402) are positioned at the top of each storage room, the blind spot in the shooting area caused by the main camera (1401) can be compensated for. In addition, when the sub-cameras (1402) are positioned at the top of each storage room, the obscuration of a newly placed object by a previously stored object can be minimized, and the refrigerator (1000) can three-dimensionally recognize an object selected by a user from various angles, enabling clear motion recognition.

[0084] Referring to 211 and 212 of FIG. 2, when the main camera (1401) is positioned at the front of the top table, the shooting area of ​​the main camera (1401) may include not only the storage compartment (refrigerator compartment) but also the door bin located inside the door (10). Accordingly, when the refrigerator (1000) detects that the door (10) is opened, it can activate the main camera (1401) to detect an object entering the storage compartment and / or the door bin.

[0085] According to one embodiment of the present disclosure, the refrigerator (1000) may recognize an object or a user's hand using only the main camera (1401), or may recognize an object or a user's hand using both the main camera (1401) and the sub cameras (1401). For example, the refrigerator (1000) may first analyze image frames acquired by the main camera (1401) to recognize an object being placed in the refrigerator (1000), an object being taken out of the refrigerator (1000), a user's hand being placed in the refrigerator (1000), and an object selected by the user. If it is difficult to accurately recognize the position of an object being placed in the refrigerator (1000) or the position of a user's hand using only the image frames acquired by the main camera (1401), the refrigerator (1000) may further utilize image frames acquired by the sub cameras (1402) to accurately recognize the position of the object or the position of the user's hand.

[0086] In FIG. 2, the main camera (1401) is positioned at the top center of the refrigerator (1000) as an example, but the present invention is not limited thereto. For example, the main camera (1401) may be positioned at the top left, top right, or center of the storage compartments of the refrigerator (1000).

[0087] FIG. 3 is a drawing for explaining a type of camera according to one embodiment of the present disclosure.

[0088] According to one embodiment of the present disclosure, at least one camera of the refrigerator (1000) may be, but is not limited to, an RGB camera, a thermal imaging camera, a depth camera, or a DVS (Dynamic vision sensor) camera.

[0089] Referring to 301 of FIG. 3, if at least one camera is an RGB camera, the refrigerator (1000) can acquire an RGB image using the RGB camera. At this time, the refrigerator (1000) can apply the RGB image to an artificial intelligence model for object recognition, thereby detecting the user's hand and food from the RGB image, and tracking the positions of the user's hand and food, respectively.

[0090] According to one embodiment of the present disclosure, when at least one camera is a depth camera, the refrigerator (1000) can more accurately recognize the storage location of food brought into the refrigerator (1000) or the movement of a user's hand.

[0091] Referring to 302 of FIG. 3, if at least one camera is a thermal imaging camera, the refrigerator (1000) can detect the heat of a user's hand within the refrigerator where a refrigerated environment is maintained through the thermal imaging camera, thereby tracking the position or movement of the user's hand. The thermal imaging camera may include an infrared sensor.

[0092] Meanwhile, if at least one camera includes a DVS camera, the refrigerator (1000) can detect the user's moving hand through the DVS camera. For example, when the user opens the door (1000) of the refrigerator (1000) and looks inside the refrigerator (1000) with only his / her eyes, the DVS camera cannot detect the user's hand. However, when the user puts his / her hand inside the refrigerator (1000) and moves it, the DVS camera can detect the user's hand. Accordingly, the refrigerator (1000) can activate only the DVS camera when the door (10) is opened to monitor whether movement occurs inside the refrigerator, and activate the main RGB camera when the movement of the user's hand is detected, thereby confirming an object selected by the user.

[0093] According to one embodiment of the present disclosure, the sub-cameras (1402) placed in each storage room may be of the same type or different types. For example, an RGB camera may be installed in the door bin and a depth camera may be installed in the drawer, but this is not a limitation.

[0094] Hereinafter, a method for determining whether an object is received or delivered by a refrigerator (1000) using at least one camera will be described in detail with reference to FIG. 4.

[0095] FIG. 4 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to determine whether to put an object in or out.

[0096] Referring to FIG. 4, a method for a refrigerator (1000) to determine whether to receive or ship an object may include steps S410 to S490. In one embodiment of the present disclosure, steps S410 to S490 may be executed by at least one processor included in the refrigerator (1000). The method for a refrigerator (1000) to determine whether to receive or ship an object is not limited to that illustrated in FIG. 4, and in one or more embodiments, steps not illustrated in FIG. 4 may be further included, or some steps may be omitted.

[0097] In step S410, the refrigerator (1000) can detect the opening or closing of the door (10). For example, the refrigerator (1000) can detect the open or closed state of the door (10) through a door opening / closing detection sensor. The door opening / closing detection sensor may include, but is not limited to, a hall sensor, a micro switch, or the like.

[0098] In step S420, the refrigerator (1000) according to one embodiment of the present disclosure may maintain the inactivation state of at least one camera included in the refrigerator (1000) if the opening of the door (10) is not detected (No in S410). That is, if the door (10) of the refrigerator (1000) is closed, at least one camera included in the refrigerator (1000) may not be activated.

[0099] In step S430, the refrigerator (1000) according to one embodiment of the present disclosure may activate at least one camera included in the refrigerator (1000) if it detects that the door (10) is open (Yes in S410). For example, if a user opens the door (10) of the refrigerator (1000), the refrigerator (1000) may turn on the operation of the main camera (1401) located at the top center. In addition, if the refrigerator (1000) further includes sub cameras (1402) in each storage compartment, the refrigerator (1000) may turn on the operation of the sub cameras (1402) together with the main camera (1401).

[0100] In step S440, the refrigerator (1000) according to one embodiment of the present disclosure may acquire at least one image frame using at least one activated camera. The refrigerator (1000) may acquire the image frames at predetermined time intervals (e.g., 0.1 second). At least one image frame may include a user's hand and / or objects (e.g., food). If the user places both hands into the refrigerator (1000), at least one image frame may include both hands.

[0101] In step S450, the refrigerator (1000) according to one embodiment of the present disclosure can detect a first object and a user's hand separately from at least one image frame. The first object may be an object whose position overlaps with the user's hand in some image frames. The first object may be an object whose position moves in some image frames. In other words, the first object may be an object being brought into or taken out of the refrigerator (1000).

[0102] According to one embodiment of the present disclosure, a refrigerator (1000) can distinguish between a user's hand and a first object (e.g., food) using an artificial intelligence model (e.g., a vision recognition model). The artificial intelligence model may be a model trained to recognize objects in images or videos and identify their locations. The artificial intelligence model may include, but is not limited to, a deep learning model such as a convolution neural network (CNN).

[0103] According to one embodiment of the present disclosure, the refrigerator (1000) can detect each of the user's hand and the first object from the image frame by applying an artificial intelligence model (e.g., a vision recognition model) to an image frame in which a user's hand is holding a first object. Then, the refrigerator (1000) can extract the user's hand by cropping the detected user's hand from the detected first object. The refrigerator (1000) can extract the first object by cropping the detected first object from the detected user's hand.

[0104] In step S460, the refrigerator (1000) according to one embodiment of the present disclosure can track the positions of each of the first object and the user's hand to determine whether to put the first object in or out.

[0105] According to one embodiment of the present disclosure, the refrigerator (1000) can track the positions of each of a first object and a user's hand by analyzing a plurality of image frames acquired through at least one camera. At this time, the refrigerator (1000) can analyze the movement directions of the user's hand and the first object, thereby determining whether the first object is being brought into the refrigerator (1000) or taken out from the refrigerator (1000).

[0106] For example, if the user's hand and the first object initially move together after the door (10) is opened, and then the movement of the first object stops within the storage compartment and only the user's hand continues to move, the refrigerator (1000) can determine that the first object has been stored. On the other hand, if the user's hand initially moves to the storage location of the first object after the door (10) is opened, and then the first object and the user's hand move together, and then the first object is no longer recognized within the storage compartment, the refrigerator (1000) can determine that the first object has been taken out of the refrigerator (1000). The operation of the refrigerator (1000) to determine whether to store or retrieve the first object will be discussed in more detail later with reference to FIG. 5.

[0107] In steps S470 and S480, the refrigerator (1000) according to one embodiment of the present disclosure may store an image of the first object, a storage location of the first object, a product name of the first object, and a date of receipt of the first object in the memory if it is determined that the first object has been received in the refrigerator (1000). For example, the refrigerator (1000) may add information related to the first object newly received or re-received in the refrigerator (1000) to an object management table (e.g., a food list).

[0108] According to one embodiment of the present disclosure, the refrigerator (1000) can generate an image of a first object from an image frame acquired through at least one camera. For example, when a user holds the first object and places it into the refrigerator (1000), a portion of the first object may be obscured by the user's hand within the image frame. In this case, the refrigerator (1000) can generate an image of the first object by removing the image of the user's hand from the image frame and then applying an inpainting technique to restore the portion of the first object that was obscured by the user's hand.

[0109] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may generate an image of the first object by cropping an image in which a portion of the first object is obscured by the user's hand. Alternatively, the refrigerator (1000) may generate an image of the first object by cropping an image that includes only the first object and not the user's hand.

[0110] According to one embodiment of the present disclosure, the refrigerator (1000) can track the movement (location) of a first object using at least one camera and identify the location where the first object stops as the storage location of the first object. The storage location of the first object may be expressed as an identification value or coordinate value of a storage compartment, but is not limited thereto.

[0111] According to one embodiment of the present disclosure, the refrigerator (1000) can obtain the product name (or product type) of the first object by applying an image frame acquired through at least one camera to an artificial intelligence model (e.g., a vision recognition model). The artificial intelligence model (e.g., a vision recognition model) may be a model that has been previously trained on millions or more food photos to infer the food name or food type. The refrigerator (1000) can also obtain the product name (or product type) of the first object by performing an optical character recognition (OCR) operation on the image frame acquired through at least one camera.

[0112] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may retrieve additional information, such as consumption amount, calories, and expiration date, related to the first object received, through an external server and additionally record the information in the object management table. Furthermore, when the refrigerator (1000) receives additional information, such as consumption amount, expiration date, and precautions, from a user related to the first object, the refrigerator (1000) may add the additional information to the object management table (e.g., food list).

[0113] In step S480, the refrigerator (1000) according to one embodiment of the present disclosure may store the shipping date of the first object in memory if it is determined that the first object has been shipped from the refrigerator (1000). For example, the refrigerator (1000) may add the shipping date of the first object to an object management table (e.g., a food list) stored in the memory.

[0114] According to one embodiment of the present disclosure, when a first object is shipped and then restocked after a certain period of time, the refrigerator (1000) can add the restocking date to an object management table (e.g., food list).

[0115] Meanwhile, according to one embodiment of the present disclosure, if the refrigerator (1000) determines that the first object has been shipped from the refrigerator (1000), it may delete information related to the first object from the object management table (e.g., food list). The refrigerator (1000) may delete information related to the first object immediately after the first object is shipped, or may delete information after a certain period of time has elapsed from the time of shipment.

[0116] Hereinafter, with reference to FIG. 5, we will look a little more closely at the operation of the refrigerator (1000) to determine whether to put in or take out the first object.

[0117] FIG. 5 is a drawing for explaining an operation of a refrigerator (1000) according to one embodiment of the present disclosure to track a user's hand or an object.

[0118] Referring to FIG. 5, the refrigerator (1000) can detect a user's hand or a first object (e.g., milk) entering and exiting the refrigerator (1000) from each image frame based on a machine learning (deep learning) algorithm. At this time, the refrigerator (1000) can set a center point for each of the user's hand and the first object, and track the movement of the user's hand or the first object based on the movement of the center point.

[0119] For example, the refrigerator (1000) can analyze image frames acquired from at least one camera to identify that the first object and the user's hand move in the same direction in the first image frame, the second image frame, the third image frame, and the fourth image frame, but that the first object stops and only the user's hand moves in the opposite direction after the fifth image frame. At this time, the refrigerator (1000) can determine that the first object has been brought into the refrigerator (1000). In addition, the refrigerator (1000) can identify the location where the first object stops as the storage location of the first object.

[0120] Although not shown in FIG. 5, the refrigerator (1000) can analyze image frames acquired from at least one camera to identify that only the user's hand moves in the first image frame, the second image frame, and the third image frame, and that the first object and the user's hand move in the same direction in the fourth image frame, the fifth image frame, and the sixth image frame. At this time, the refrigerator (1000) can determine that the first object has been released from the refrigerator (1000).

[0121] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may acquire tracking graphs in the process of tracking the movements of the user's hand and the first object. For example, referring to FIG. 5, the refrigerator (1000) may acquire a tracking graph (510) of the user's hand and a tracking graph (520) of the first object. The refrigerator (1000) may compare the tracking graph (510) of the user's hand and the tracking graph (520) of the first object to determine whether the first object has been placed in the refrigerator (1000).

[0122] Meanwhile, according to one embodiment of the present disclosure, when the refrigerator (1000) determines that the first object has been received, it may store an image of the first object, a storage location of the first object, a product name of the first object, a date of receipt of the first object, additional information (e.g., calories, expiration date, consumption amount, etc.), etc., in a table format in the memory. For example, refer to FIG. 6.

[0123] FIG. 6 is a drawing for explaining an object management table (600) according to one embodiment of the present disclosure.

[0124] In Fig. 6, the storage space corresponding to the upper left door is defined as compartment A, the storage space corresponding to the upper right door is defined as compartment B, the storage space corresponding to the lower left door is defined as compartment C, and the storage space corresponding to the lower right door is defined as compartment D, as an example. In addition, in Fig. 6, the left drawer is defined as drawer 1, the right drawer is defined as drawer 2, the storage rooms are divided into rows 1, 2, and 3 from the bottom, and the door bins are divided into rows 1, 2, and 3 from the bottom, as an example.

[0125] If the refrigerator (1000) determines that cabbage has been received, it can add the date of receipt of the cabbage (e.g., 2023.07.04), product classification (e.g., vegetable-cabbage), storage location (e.g., B compartment drawer 1), and image of the cabbage to the object management table (600). The product classification may be obtained by the refrigerator (1000) through AI vision recognition. The storage location may be obtained by the refrigerator (1000) by tracking the movement of the cabbage. The image of the cabbage may be an image directly generated by the refrigerator (1000) through at least one camera, or may be a pre-stored standard image or icon image.

[0126] Additionally, the refrigerator (1000) may add the calories of cabbage (e.g., 31 Kcal per 100 g) to the object management table (600). At this time, the calories of cabbage may be obtained from an external server or input by a user.

[0127] If the refrigerator (1000) determines that milk has been received, it can add the date of receipt (e.g., 2023.07.03), product classification (e.g., dairy products - milk), storage location (e.g., section A, column 2), and an image of the milk to the object management table (600). In addition, if the milk has been released and then re-released, the refrigerator (1000) can record the date of release (e.g., 2023.07.09 1 PM) and the date of re-release (e.g., 2023.07.09 1:30 PM) in the object management table (600).

[0128] The refrigerator (1000) may also record the milk consumption (e.g., 50%) upon restocking. The milk consumption may be input by the user or may be acquired by the refrigerator (1000) via at least one camera. For example, if the milk bottle is transparent, the refrigerator (1000) may detect changes in the image of the milk bottle via the camera upon restocking, thereby determining the milk consumption.

[0129] The refrigerator (1000) may also record the expiration date of the milk (e.g., 2023.07.17) in the object management table (600). For example, the refrigerator (1000) may obtain the expiration date of the milk from an image of the milk and automatically enter it into the object management table (600).

[0130] According to one embodiment of the present disclosure, when the storage location of milk changes upon restocking, the refrigerator (1000) may record the location at the time of restocking as the storage location of the milk. Meanwhile, the refrigerator may also update the image of the milk upon restocking.

[0131] If the refrigerator (1000) determines that eggs have been received, it can add the date of receipt of the eggs (e.g., 2023.07.01), product classification (e.g., fresh food - eggs), expiration date (e.g., 2023.08.01), storage location (e.g., section A, column 1), and image of the eggs to the object management table (600). The expiration date of the eggs may be input by the user.

[0132] According to one embodiment of the present disclosure, the storage location of the object management table (600) can be utilized to recognize an object selected by a user by comparing it with the location of the user's hand. Below, a method for the refrigerator (1000) to identify an object selected by a user among objects stored in the refrigerator (1000) will be described in detail with reference to FIG. 7.

[0133] FIG. 7 is a flowchart for explaining an object identification method of a refrigerator (1000) according to one embodiment of the present disclosure.

[0134] Referring to FIG. 7, the object identification method of the refrigerator (1000) may include steps S710 to S740. In one embodiment of the present disclosure, steps S710 to S740 may be executed by at least one processor included in the refrigerator (1000). The object identification method of the refrigerator (1000) is not limited to that illustrated in FIG. 7, and in one or more embodiments, steps not illustrated in FIG. 7 may be further included, or some steps may be omitted.

[0135] In step S710, the refrigerator (1000) according to one embodiment of the present disclosure can track the position of a user's hand placed in the refrigerator (1000) using at least one camera.

[0136] According to one embodiment of the present disclosure, the refrigerator (1000) can activate at least one camera when detecting an open state of the door through a door open / close detection sensor. The refrigerator (1000) can then acquire image frames through the activated at least one camera.

[0137] The refrigerator (1000) can recognize a user's hand from each of the image frames acquired through at least one camera. For example, the refrigerator (1000) can recognize a user's hand by sequentially inputting each image frame into an artificial intelligence model (vision recognition model) that has learned images containing various users' hands.

[0138] The refrigerator (1000) can track the position of the user's hand based on the center point of the user's hand recognized in each image frame. For example, the refrigerator (1000) can set the center point of the user's hand in each image frame and track the position of the user's hand based on the movement of the center point. For example, the refrigerator (1000) can track the movement of the user's hand from column 1 of section A to column 3 of section B.

[0139] According to one embodiment of the present disclosure, the refrigerator (1000) can identify the initial position or the final position of the user's hand. For example, if the refrigerator (1000) tracks the movement of the user's hand from column 1 of compartment A to column 3 of compartment B, the refrigerator (1000) can identify column 1 of compartment A as the initial position of the user's hand and column 3 of compartment B as the final position of the user's hand. In addition, the refrigerator (1000) can also identify the transit position of the user's hand. For example, if the user's hand moves from column 1 of compartment A to column 3 of compartment B and then from column 3 of compartment B to drawer 1 of compartment A, the refrigerator (1000) can identify column 1 of compartment A as the initial position of the user's hand, column 3 of compartment B as the transit position of the user's hand, and drawer 1 of compartment A as the final position of the user's hand.

[0140] According to one embodiment of the present disclosure, when a user puts both hands into a refrigerator (1000) at the same time, the refrigerator (1000) may apply image frames to an artificial intelligence model to track the positions of each of the hands.

[0141] In step S720, the refrigerator (1000) according to one embodiment of the present disclosure can compare the position of the user's hand with storage location information of objects stored in the memory.

[0142] According to one embodiment of the present disclosure, as illustrated in FIG. 6, the refrigerator (1000) can store and manage the storage locations of objects in memory when they are stored. Accordingly, the refrigerator (1000) can compare the location of the user's hand, currently tracked through at least one camera, with the storage locations of objects stored in memory.

[0143] In step S730, the refrigerator (1000) according to one embodiment of the present disclosure can identify at least one object selected by the user among objects stored in the refrigerator (1000) based on a result of comparing the position of the user's hand with the storage location information of objects stored in the memory.

[0144] According to one embodiment of the present disclosure, the refrigerator (1000) can identify an object stored in a location overlapping the location of the user's hand as an object selected by the user, based on a comparison between the location of the user's hand and the storage locations of objects. In particular, according to one embodiment of the present disclosure, the refrigerator (1000) can identify an object stored in a location overlapping the location of the user's hand for a predetermined period of time (e.g., 3 seconds) or longer as an object selected by the user. For example, if the user's hand is positioned in drawer 2 of compartment B for 3 seconds or longer and cabbage is stored in drawer 2 of compartment B, the refrigerator (1000) can identify that the user has selected cabbage. Additionally, if the user's hand is positioned in drawer 2 of compartment B where cabbage is stored for 3 seconds and then also in row 2 of compartment A where strawberries are stored for 3 seconds, the refrigerator (1000) can identify that the user has selected cabbage and strawberries.

[0145] According to one embodiment of the present disclosure, the refrigerator (1000) can identify at least one object selected by the user among objects stored in the refrigerator (1000) by further considering the motion of the user's hand. That is, if the refrigerator (1000) detects a preset motion (e.g., holding for a long time or touching twice or more) for an object stored at a location overlapping with the location of the user's hand, the refrigerator (1000) can identify the object as an object selected by the user. For example, if the user's hand is located in Drawer 2 of compartment B where cabbages are stored and the user touches the cabbage twice or more, the refrigerator (1000) can identify that the user has selected the cabbage.

[0146] According to one embodiment of the present disclosure, the refrigerator (1000) can identify at least one object selected by the user among objects stored in the refrigerator (1000) by further considering the shape of the user's hand. That is, if the refrigerator (1000) detects a preset shape of the user's hand (e.g., extending only the second finger, making a fist, making a V sign, etc.) for an object stored in a location overlapping with the location of the user's hand, the refrigerator (1000) can identify the object as an object selected by the user. For example, if the user's hand is located in the second row of compartment A where milk is stored and touches the milk with only the second finger extended, the refrigerator (1000) can identify that the user has selected milk. In addition, if the user's hand moves to the first row of compartment A where eggs are stored and makes a V sign with the second and third fingers, the refrigerator (1000) can identify that the user has selected eggs.

[0147] According to one embodiment of the present disclosure, when the refrigerator (1000) identifies at least one object selected by a user, the refrigerator can add the object to the selected product list (100). The selected product list (100) can be stored and managed in a storage standby database (e.g., a temporary storage DB). The storage standby database (e.g., a temporary storage DB) can be a DB that stores history only for a certain period of time (e.g., 10 minutes) after the user selects an object.

[0148] In step S740, the refrigerator (1000) according to one embodiment of the present disclosure may provide information related to at least one object selected by the user. For example, the refrigerator (1000) may provide information related to an object added to the selected product list (100).

[0149] According to one embodiment of the present disclosure, information related to at least one object selected by the user may include, but is not limited to, at least one of a storage location of the object, an image of the object, an expiration date of the object, calories of the object, a consumption amount of the object, a degree of spoilage of the object, a degree of discoloration of the object, a price of the object, or an appropriate consumption period of the object.

[0150] According to one embodiment of the present disclosure, the refrigerator (1000) can retrieve information related to at least one object selected by the user from an object management table (600) (e.g., a food list) stored in memory. Furthermore, the refrigerator (1000) can also request and receive information related to at least one object selected by the user from an external server (e.g., a home appliance management server).

[0151] According to one embodiment of the present disclosure, the refrigerator (1000) may provide information related to at least one object selected by a user audibly through a speaker or visually through a display unit (1300). The display unit (1300) may include a door display included on the outer surface of the door (10).

[0152] According to one embodiment of the present disclosure, the refrigerator (1000) can output information related to at least one object through a speaker as a voice when the door (10) is open. When the door (10) of the refrigerator (1000) is open, it may be difficult for the user to check the door display. Therefore, the refrigerator (1000) can provide convenience to the user by providing information related to an object selected by the user through a voice when the door (10) is open. For example, the refrigerator (1000) can output the expiration date of the selected object, the calories of the selected object, the consumption amount of the selected object, the appropriate eating period of the selected object, the degree of discoloration of the selected object, the degree of decay of the selected object, etc. through a voice.

[0153] According to one embodiment of the present disclosure, the refrigerator (1000) may output a notification sound through the speaker of the refrigerator (1000) upon identifying at least one object selected by the user. For example, the refrigerator (1000) may notify the user that the object has been added to the selected product list (100) through a notification sound (or voice). The operation of the refrigerator (1000) outputting a voice will be discussed in more detail later with reference to FIG. 8.

[0154] According to one embodiment of the present disclosure, when the door (10) is switched from an open state to a closed state, the refrigerator (1000) can output information related to at least one object selected by the user on the display unit (1300). For example, the refrigerator (1000) can output a selected product list (100) including an image of the selected object and a storage location of the selected object on the display unit (1300) of the refrigerator (1000). The selected product list (100) may be provided in the form of a table or may be provided in the form of an internal image of the refrigerator (1000).

[0155] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may further output at least one of expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, or appropriate eating period information as information related to at least one object selected by the user, on the display unit (1300) of the refrigerator (1000).

[0156] According to one embodiment of the present disclosure, the refrigerator (1000) tracks the position of a user's hand to accurately identify an object selected by the user and provides information related to the object selected by the user, thereby enabling the user to quickly obtain information about the object stored within the refrigerator (1000). Below, with reference to FIGS. 8 to 9A, the operation of the refrigerator (1000) providing information related to the object selected by the user will be further examined.

[0157] FIG. 8 is a drawing for explaining an operation of a refrigerator (1000) according to one embodiment of the present disclosure to provide information related to an object through voice.

[0158] Referring to FIG. 8, when a user opens the door (10) of a refrigerator (1000) and moves his / her hand to a location where tangerines are stored, the refrigerator (1000) can activate the main camera (1401) to recognize the user's hand and track the location of the user's hand. When the user's hand touches a tangerine, the refrigerator (1000) can determine that the user has selected a tangerine by comparing the location of the user's hand with the storage locations of the objects.

[0159] According to one embodiment of the present disclosure, if the refrigerator (1000) determines that the user has selected a tangerine, it may output information related to the tangerine through voice. For example, the refrigerator (1000) may output status information (801) through voice, such as, "The tangerine you selected was delivered in November 2023 and is currently browning."

[0160] According to one embodiment of the present disclosure, the refrigerator (1000) can detect the degree of spoilage or discoloration of stored products using a gas sensor, an infrared sensor, and a camera, and provide information on the degree of spoilage or discoloration. Through this, the refrigerator (1000) can quickly inform the user of the risk of ingestion of stored products.

[0161] Additionally, if the refrigerator (1000) determines that the user has selected a tangerine, it may output a notification sound (e.g., a beep) (802) or a voice (e.g., a tangerine has been selected) while adding the tangerine to the selected product list (100). In this case, the user can confirm whether the desired object has been selected correctly through the notification or voice.

[0162] According to one embodiment of the present disclosure, when outputting a notification sound or voice guidance, the refrigerator (1000) can visually indicate to the user that a notification sound has been output by adjusting the color of the internal lighting. For example, the refrigerator (1000) can blink an LED display provided inside the refrigerator (1000) in a predetermined color (e.g., red).

[0163] FIG. 9a is a drawing for explaining an operation of a refrigerator (1000) according to one embodiment of the present disclosure to provide a list of selected products (100).

[0164] Referring to FIG. 9a, a user can open the door (10) of a refrigerator (1000), select objects, and then close the door (10). At this time, the refrigerator (1000) can output information related to the objects selected by the user on the display unit (1300).

[0165] According to one embodiment of the present disclosure, the refrigerator (1000) may display an image of an object and a storage location of the object as information related to an object selected by a user. For example, referring to screen 910 of FIG. 9A , the refrigerator (1000) may display a selection product list (100) (hereinafter, also referred to as a selection list) that displays images of objects at their storage locations. The images of the objects may be displayed on an actual internal image of the refrigerator (1000).

[0166] The user can confirm the selected object through the selection product list (100) displayed on the display unit (1300). If the user selects an object incorrectly, the user can delete the incorrectly selected object from the selection product list (100). That is, if the refrigerator (1000) receives an input from the user to cancel the selection of an object, the object can be deleted from the selection product list (100). For example, if the user incorrectly selects a cabbage, the user can touch the image (901) of the cabbage. When the user touches the image (901) of the cabbage, an icon (e.g., X) for canceling the selection may be displayed, and when the user selects the icon, the image (901) of the cabbage may disappear from the product selection list.

[0167] In Fig. 9a, an example is given where only images of objects selected by the user are displayed in the selection product list (100), but the present invention is not limited thereto. Refer to Fig. 9b.

[0168] FIG. 9b is a drawing for explaining editing of a selection product list (100) according to one embodiment of the present disclosure.

[0169] Referring to FIG. 9b, the refrigerator (1000) can display images of all objects stored in the refrigerator (1000) on the display unit (1300). Images of all objects can be displayed at the storage locations of each object. For example, if milk is stored at the left end of the first row, the image of the milk can be displayed at the left end of the first row, and if carrots are stored at the right end of the first row, the image of the carrots can be displayed at the right end of the first row.

[0170] According to one embodiment of the present disclosure, the refrigerator (1000) can display objects selected by the user in a manner that distinguishes them from other objects. For example, the refrigerator (1000) can clearly display objects selected by the user among all objects and dimly display the remaining objects. The refrigerator (1000) can also display objects selected by the user among all objects as color images and display the remaining images as black and white images. The refrigerator (1000) can also add a specific icon (e.g., a star icon) to the images of objects selected by the user among all objects.

[0171] The user can confirm the selected object through the selection product list (100) displayed on the display unit (1300). If the object selected by the user is incorrectly displayed, the user can change the object selection. For example, referring to 920 of FIG. 9b, if the user selects an egg, but the refrigerator (1000) incorrectly identifies the user as selecting a carrot, the user can deselect the carrot in the selection product list (100) and select the egg. If the refrigerator (1000) receives a user input for deselecting the carrot and selecting the egg, the refrigerator can display the egg instead of the carrot in the selection product list (100) to be distinct from other objects. For example, referring to 930 of FIG. 9b, the carrot may be displayed blurry because it has been deselected, and the egg may be displayed clearly.

[0172] Therefore, according to one embodiment of the present disclosure, a user can easily change the selection of objects in the selection product list (100).

[0173] In FIGS. 9A and 9B, the selection product list (100) is described as an example in which images of objects are displayed on the internal image of the refrigerator (1000), but the present invention is not limited thereto. The selection product list (100) may also be displayed in the form of a table including the product names of the objects.

[0174] Hereinafter, a method in which a refrigerator (1000) provides a user with additional information related to a selected object using a selection product list (100) will be described with reference to FIGS. 10 to 13.

[0175] FIG. 10 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to provide recipe information.

[0176] Referring to FIG. 10, a method for a refrigerator (1000) to provide recipe information may include steps S1010 to S1030. In one embodiment of the present disclosure, steps S1010 to S1030 may be executed by at least one processor included in the refrigerator (1000). The method for a refrigerator (1000) to provide recipe information is not limited to that illustrated in FIG. 10, and in one or more embodiments, additional steps not illustrated in FIG. 10 may be included, or some steps may be omitted.

[0177] In step S1010, a refrigerator (1000) according to one embodiment of the present disclosure can output a selection product list (100) including at least one object selected by a user.

[0178] For example, when the refrigerator (1000) identifies an object selected by the user from among stored objects by tracking the position of the user's hand, the refrigerator can add the object to the selected product list (100), and when the door (10) is switched to a closed state, the refrigerator can display the selected product list (100) on the display unit (1300).

[0179] In step S1020, a refrigerator (1000) according to one embodiment of the present disclosure may receive a user input for selecting a plurality of objects included in a selection product list (100). The user input may be predefined. According to one embodiment of the present invention, the user input for selecting a plurality of objects may be a touch input or a voice input.

[0180] For example, the refrigerator (1000) may receive a user input of touching the image of the first object, the image of the second object, and the image of the third object among the images of objects included in the selection product list (100) for a predetermined period of time or longer. Alternatively, the refrigerator (1000) may receive an input of touching the image of the first object, the image of the second object, and the image of the third object among the images of objects included in the selection product list (100) two or more times. Alternatively, the refrigerator (1000) may receive a user voice through a microphone to select the first object, the second object, and the third object.

[0181] In step S1030, the refrigerator (1000) according to one embodiment of the present disclosure can provide recipe information corresponding to a combination of multiple objects.

[0182] For example, if a user selects a first object, a second object, and a third object from a selection product list (100), the refrigerator (1000) can display recipe information for a recommended dish that can be completed by combining the first object, the second object, and the third object on the display unit (1300).

[0183] According to one embodiment of the present disclosure, the refrigerator (1000) can generate recipe information for a recommended dish using a recommended dish algorithm stored in memory. For example, at least one processor of the refrigerator (1000) can input a first object, a second object, and a third object into the recommended dish algorithm to obtain recipe information for the recommended dish.

[0184] According to one embodiment of the present disclosure, the refrigerator (1000) may obtain recipe information corresponding to a combination of multiple objects from an external server. For example, when a user selects multiple objects from a selection product list (100), the refrigerator (1000) may request recipe information while transmitting identification information regarding the multiple objects to an external server (e.g., a home appliance management server). The external server may search for recipe information corresponding to a combination of multiple objects among various previously stored recipe information. The external server may transmit recipe information corresponding to the searched combination of multiple objects to the refrigerator (1000).

[0185] According to one embodiment of the present disclosure, if there are multiple dishes that can be completed by combining multiple objects selected by the user, the refrigerator (1000) may provide multiple recipe information. For example, if the user selects chicken and cabbage, the refrigerator (1000) may provide recipe information for chicken salad and recipe information for dakgalbi (spicy stir-fried chicken).

[0186] FIG. 11 is a drawing for explaining an operation of a refrigerator (1000) according to one embodiment of the present disclosure to provide additional information including recipe information.

[0187] Referring to screen 1101 of FIG. 11, the refrigerator (1000) can display a selection product list (100) including images of objects selected by a user on the display unit (1300). At this time, the refrigerator (1000) can receive an input from the user to select multiple objects from the selection product list (100). For example, the refrigerator (1000) can receive a user input to select an image of cabbage after selecting an image of milk.

[0188] Referring to screen 1102 of FIG. 11, the refrigerator (1000) can obtain and provide information related to milk and cabbage selected by the user from the object management table (600). For example, the refrigerator (1000) can provide the calories of milk (e.g., 40 Kcal per 100 ml), the date of receipt of milk (e.g., 2022.04. 23), the calories of cabbage (e.g., 31 Kcal per 100 g), the date of receipt of cabbage (2202.04. 23), the date of consumption of cabbage (e.g., 2022.4. 24), the amount of cabbage consumed (50%), etc. The user can check the information related to milk and cabbage and consume the milk and cabbage at an appropriate time.

[0189] Additionally, the refrigerator (1000) can recommend cabbage juice, a combination of milk and cabbage, as a recipe, and can also provide recipe information for the cabbage juice. Accordingly, the user can gain ideas for a variety of dishes that can be made using the foods stored in the refrigerator (1000).

[0190] FIG. 12 is a flowchart illustrating a method for providing purchase information by a refrigerator (1000) according to one embodiment of the present disclosure.

[0191] In step S1210, a refrigerator (1000) according to one embodiment of the present disclosure can output a selection product list (100) including at least one object selected by a user.

[0192] For example, when the refrigerator (1000) identifies an object selected by the user from among stored objects by tracking the position of the user's hand, the refrigerator can add the object to the selected product list (100), and when the door (10) is switched to a closed state, the refrigerator can display the selected product list (100) on the display unit (1300).

[0193] In step S1220, the refrigerator (1000) according to one embodiment of the present disclosure may receive a user input requesting a purchase of a second object included in the selection product list (100). According to one embodiment of the present invention, the user input requesting a purchase of the second object may be a touch input or a voice input.

[0194] According to one embodiment of the present disclosure, the second object for which a user requests a purchase may be one or multiple. For example, the user may enter a purchase request for milk, eggs, and carrots among the objects included in the selection product list (100).

[0195] In step S1230, the refrigerator (1000) according to one embodiment of the present disclosure may request purchase information for a second object from an external server (2000).

[0196] A refrigerator (1000) according to one embodiment of the present disclosure may request purchase information regarding a second object from an external server (2000) via wireless communication (e.g., Wi-Fi communication). The external server (2000) may be a server that manages home appliances or a server that relays online shopping, but is not limited thereto.

[0197] In step S1240, an external server (2000) according to one embodiment of the present disclosure can search for purchase information for a second object.

[0198] According to one embodiment of the present disclosure, an external server (2000) may retrieve purchase information for a second object in response to a request from a refrigerator (1000). The external server (2000) may retrieve purchase information for the second object from among the purchase information for previously stored objects. The external server (2000) may also collect purchase information for the second object from an online mart website. The purchase information may include, but is not limited to, the purchase site (seller), purchase price, shipping cost, minimum purchase quantity, discount event, etc.

[0199] In step S1250, an external server (2000) according to one embodiment of the present disclosure may transmit purchase information for a second object to a refrigerator (1000).

[0200] According to one embodiment of the present disclosure, when the search for purchase information for the second object is completed, the external server (2000) can transmit the purchase information for the second object to the refrigerator (1000) via wireless communication.

[0201] In step S1260, the refrigerator (1000) according to one embodiment of the present disclosure may provide purchase information for the second object.

[0202] For example, the refrigerator (1000) may display price information, shipping costs, minimum purchase quantity, discount events, etc. for each purchase site (seller) from which the second object can be purchased on the display unit (1300). The user may check purchase information for the second object and complete the purchase.

[0203] Accordingly, according to one embodiment of the present disclosure, a user can check the food inventory within the refrigerator (1000) and immediately purchase any food items that are in short supply. Referring to FIG. 13, the operation of the refrigerator (1000) providing purchase information through the display unit (1300) will be further examined.

[0204] FIG. 13 is a drawing for explaining an operation of a refrigerator (1000) providing purchase information according to one embodiment of the present disclosure.

[0205] Referring to screen 1301 of FIG. 13, a refrigerator (1000) according to one embodiment of the present disclosure can display a selection product list (100) including at least one object selected by a user on a display unit (1300). For example, if a user selects milk, eggs, and carrots in the refrigerator (1000), a selection product list (1300) including an image of milk, an image of an egg, and an image of a carrot can be displayed on the display unit (1300).

[0206] A user can select an object from the list of items to purchase (1300) and add it to their shopping cart. For example, if the user determines that they need to repurchase eggs and carrots, they can add them to their shopping cart.

[0207] Meanwhile, users can check the purchase information for eggs before adding them to their shopping cart. For example, users can request purchase information about eggs from the refrigerator (1000) by long-pressing the image of an egg before adding it to their shopping cart.

[0208] Referring to screen 1302 of FIG. 13, when a request for purchase information on eggs is received from a user, the refrigerator (1000) may obtain purchase information on eggs from an external server (2000) and display the information on the display unit (1300). For example, the refrigerator (1000) may provide price information on eggs by purchase site (seller). The user may check the price information on eggs by purchase site (seller) and proceed with the purchase of eggs from the desired purchase site.

[0209] According to one embodiment of the present disclosure, the refrigerator (1000) may display price information regarding previous purchases by the user. For example, if the user has a history of purchasing eggs on April 23, 2023, the refrigerator (1000) may display the price (e.g., KRW 3,000) of the eggs purchased on April 23, 2023. At this time, the user may compare the price information of the eggs previously purchased with the current price information.

[0210] Meanwhile, according to one embodiment of the present disclosure, a user may select multiple products from a selection product list (100) and purchase multiple products at once. According to one embodiment of the present disclosure, when a user selects multiple products, the refrigerator (1000) may recommend a purchase site (seller) to the user based on criteria set by the user (e.g., lowest price product or fastest delivery date), taking into account shipping costs, etc. In this case, the refrigerator (1000) may directly determine the recommended purchase site, or may determine the recommended purchase site by linking with an external server (2000).

[0211] According to one embodiment of the present disclosure, the refrigerator (1000) may provide purchase information on a second object directly without displaying a list of selected products, based on a user gesture detected within the refrigerator (1000). For example, the user may open the door (10), select an egg within the refrigerator (1000), and then make a specific gesture (e.g., shaking the egg left and right) to request a purchase of the egg. At this time, the refrigerator (1000) may detect the user's specific gesture and, when the door (10) is switched to a closed state, display purchase information on the egg on the display unit (1300). Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may provide purchase information on the egg through a user terminal (e.g., a mobile phone) connected via a server in addition to the display unit (1300).

[0212] Hereinafter, a method for a refrigerator (1000) to perform a control command related to a selected object according to a user's gesture for the selected object will be described in detail with reference to FIG. 14.

[0213] FIG. 14 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to perform a control command according to a user's gesture.

[0214] Referring to FIG. 14, a method for a refrigerator (1000) to perform a control command according to a user's gesture may include steps S1410 to S1430. In one embodiment of the present disclosure, steps S1410 to S1430 may be executed by at least one processor included in the refrigerator (1000). The method for a refrigerator (1000) to provide recipe information is not limited to that illustrated in FIG. 14, and in one or more embodiments, steps not illustrated in FIG. 14 may be further included, or some steps may be omitted.

[0215] In step S1410, the refrigerator (1000) according to one embodiment of the present disclosure can track the position of the user's hand and identify at least one object selected by the user among objects stored in the refrigerator (1000). For example, when the open state of the door (10) is detected, the refrigerator (1000) can activate at least one camera (e.g., the main camera (1401) or sub cameras (1402)) and track the position of the user's hand using the at least one camera. In addition, the refrigerator (1000) can compare the position of the user's hand with the storage positions of the objects stored in the refrigerator (1000) to identify at least one object selected by the user.

[0216] Step S1410 corresponds to steps S710 to S730 of FIG. 7, so redundant descriptions will be omitted.

[0217] In step S1420, a refrigerator (1000) according to one embodiment of the present disclosure may detect a user gesture for at least one object selected by the user. The user gesture may be predefined. The user gesture may be pre-matched with a control command. For example, a gesture of double-touching an object may be pre-matched with a control command to add the object to a product selection list (100).

[0218] According to one embodiment of the present disclosure, the refrigerator (1000) can acquire multiple image frames including an object and a user's hand through at least one camera. The refrigerator (1000) can separate and recognize the object and the user's hand in each image frame. The refrigerator (1000) can analyze the multiple image frames to detect the shape, movement, etc. of the user's hand. The refrigerator (1000) can detect the user's gesture based on the shape or movement of the user's hand.

[0219] In step S1430, the refrigerator (1000) according to one embodiment of the present disclosure can perform a control command related to at least one object according to a user's gesture.

[0220] Control commands according to one embodiment of the present disclosure may include, but are not limited to, providing purchase information, adding a selected object to a list of selected products, providing recipe information, and providing expiration date information by voice.

[0221] According to one embodiment of the present disclosure, when a refrigerator (1000) detects a user's gesture, it can identify a control command corresponding to the user's gesture. Then, the refrigerator (1000) can execute the control command corresponding to the user's gesture.

[0222] For example, the refrigerator (1000) can detect a gesture in which a user spreads two fingers in front of an egg. In response to the gesture of spreading two fingers, the refrigerator (1000) can output expiration date information of the egg as a voice. Here, the expiration date may include an expiration date entered by the manufacturer or an expiration date input by the user. Therefore, according to one embodiment of the present disclosure, a user can easily obtain information about a specific object through a simple gesture.

[0223] With reference to FIG. 15, let us take a closer look at the operation of the refrigerator (1000) performing a control command related to at least one object according to a user's gesture.

[0224] FIG. 15 is a drawing for explaining an operation in which a refrigerator (1000) according to one embodiment of the present disclosure performs a control command according to a user's gesture.

[0225] Referring to Table 1510 of FIG. 15, a first gesture that involves repeating a finger movement (e.g., a double tap), a second gesture that involves touching an object with a finger for a certain amount of time, and a third gesture that involves holding an object and rotating the hand clockwise or counterclockwise may be matched to a control command that links to a purchase site for the object.

[0226] Accordingly, according to one embodiment of the present disclosure, the refrigerator (1000) can store juice in the selected product list when the user holds the juice bottle in the hand for a predetermined period of time (e.g., 1 second) or detects a gesture of double-touching the juice bottle with a finger. At this time, the selected product list can be stored in a storage standby database (temporary storage DB). The storage standby database can be a database that stores history only for a predetermined period of time (e.g., 10 minutes) after object selection.

[0227] According to one embodiment of the present disclosure, when the refrigerator (1000) detects a gesture of a user holding a juice bottle in his hand and turning it twice clockwise or counterclockwise, the refrigerator can perform an action of directly connecting to a juice purchase site.

[0228] Therefore, according to one embodiment of the present disclosure, the refrigerator (1000) can recognize the user's gestures and provide various information to the user, so that it can be utilized for various food consumption.

[0229] FIG. 16 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to output a notification message as a voice.

[0230] Referring to FIG. 16, a method for a refrigerator (1000) to output a notification message as a voice may include steps S1610 to S1630. In one embodiment of the present disclosure, steps S1610 to S1630 may be executed by at least one processor included in the refrigerator (1000). The method for a refrigerator (1000) to output a notification message as a voice is not limited to that illustrated in FIG. 16, and in one or more embodiments, steps not illustrated in FIG. 16 may be further included, or some steps may be omitted.

[0231] In step S1610, the refrigerator (1000) according to one embodiment of the present disclosure can track the position of the user's hand and identify at least one object selected by the user among objects stored in the refrigerator (1000). For example, when the open state of the door (10) is detected, the refrigerator (1000) can activate at least one camera (e.g., the main camera (1401) or sub cameras (1402)) and track the position of the user's hand using the at least one camera. In addition, the refrigerator (1000) can identify at least one object selected by the user by comparing the position of the user's hand with the storage positions of the objects stored in the refrigerator (1000).

[0232] Step S1610 corresponds to steps S710 to S730 of FIG. 7, so redundant descriptions will be omitted.

[0233] In step S1620, the refrigerator (1000) according to one embodiment of the present disclosure can determine whether there is a notification message previously output through the user terminal (or external device) in relation to at least one object selected by the user.

[0234] According to one embodiment of the present disclosure, the previously output notification message may include, but is not limited to, a message that the expiration date (or use-by date, consumption-by date) has passed, a message that the expiration date is approaching, a message that caution is required when consuming in relation to the user's health (e.g., diabetes, allergies), a message that consumption is strongly recommended in relation to the user's health (e.g., osteoporosis), etc.

[0235] According to one embodiment of the present disclosure, when an external server (2000) managing a refrigerator (1000) outputs a notification message related to an object stored in the refrigerator (1000) through a user terminal, the external server (2000) can transmit information to the refrigerator (1000) that a notification message has been output related to an object stored in the refrigerator. At this time, the refrigerator (1000) can store identification information of the object for which the notification message has been output and the contents of the notification message.

[0236] In addition, when the refrigerator (1000) identifies an object selected by the user, it can search for a history of notification messages output via the user terminal related to the object selected by the user. For example, when it identifies that the user selected milk, the refrigerator (1000) can search for a history of notification messages output via the user terminal related to milk.

[0237] In step S1630, the refrigerator (1000) according to one embodiment of the present disclosure may output a notification message as a voice through a speaker if there is a notification message previously output through the user terminal in relation to at least one object selected by the user (Yes in S1630).

[0238] According to one embodiment of the present disclosure, the refrigerator (1000) may, when adding an object selected by a user to the selected product list (100), re-output a previously outputted notification message through a speaker in relation to the object through a user terminal. For example, if a user has a crustacean allergy and a notification message warning of allergy to crab has been output through the user terminal in the past, the refrigerator (1000) may output a notification message warning of allergy in a voice format upon detecting that the user has selected crab.

[0239] In this case, the user can recall the contents of a notification message that the user terminal had not yet checked or had forgotten in relation to the selected object.

[0240] Below, with reference to FIG. 17, we will look a little more closely at the operation of the refrigerator (1000) outputting a notification message as a voice.

[0241] FIG. 17 is a drawing for explaining an operation of a refrigerator (1000) according to one embodiment of the present disclosure to output a notification message as a voice.

[0242] Referring to FIG. 17, the refrigerator (1000) can activate at least one camera (e.g., the main camera (1401)) when the door (10) is opened and track the position of the user's hand using the at least one camera. At this time, the refrigerator (1000) can identify that the user is selecting a tangerine by comparing the position of the user's hand with the storage locations of objects.

[0243] If the refrigerator (1000) has previously output a notification message through the user terminal (or external device) regarding the tangerines selected by the user, the refrigerator (1000) may notify the user again via voice. For example, the refrigerator (1000) may output a notification message (1701) stating, "The expiration date of the selected product has passed." Furthermore, the refrigerator (1000) may prompt the user to take the next action by outputting a voice question such as, "Would you like to repurchase?"

[0244] For example, the refrigerator (1000) may receive a voice (1702) from the user saying "Connect me to a repurchase" through the microphone. The refrigerator (1000) may analyze the received voice using natural language processing technology and then provide repurchase information for tangerines to the display unit (1300).

[0245] According to one embodiment of the present disclosure, when outputting a notification message by voice, the refrigerator (1000) can visually indicate to the user that a notification message is being output by adjusting the color of the internal lighting. For example, the refrigerator (1000) can blink an LED display provided inside the refrigerator (1000) in a predetermined color (e.g., red).

[0246] Meanwhile, while the refrigerator (1000) has described an embodiment in which the refrigerator identifies an object selected by the user by comparing the position of the user's hand with the storage locations of the objects, the present invention is not limited thereto. According to one embodiment of the present disclosure, the refrigerator (1000) may also identify an object selected by the user by analyzing an image frame at the time a specific gesture is detected. Hereinafter, with reference to FIG. 18, a method for the refrigerator (1000) to identify an object selected by the user without comparing the position of the user's hand with the storage locations of the objects will be described in detail.

[0247] FIG. 18 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to identify at least one object selected by a user.

[0248] Referring to FIG. 18, the object identification method of the refrigerator (1000) may include steps S1810 to S1840. In one embodiment of the present disclosure, steps S1810 to S1840 may be executed by at least one processor included in the refrigerator (1000). The object identification method of the refrigerator (1000) is not limited to that illustrated in FIG. 18, and in one or more embodiments, steps not illustrated in FIG. 18 may be further included, or some steps may be omitted.

[0249] In step S1810, the refrigerator (1000) according to one embodiment of the present disclosure can track a user's hand entering the refrigerator (1000) using at least one camera.

[0250] According to one embodiment of the present disclosure, the refrigerator (1000) can activate at least one camera when detecting an open state of the door through a door open / close detection sensor. The refrigerator (1000) can then acquire image frames through the activated at least one camera.

[0251] The refrigerator (1000) can recognize a user's hand from each of the image frames acquired through at least one camera. For example, the refrigerator (1000) can recognize a user's hand by sequentially inputting each image frame into an artificial intelligence model (vision recognition model) that has learned images containing various users' hands.

[0252] In step S1820, the refrigerator (1000) according to one embodiment of the present disclosure may detect a specific gesture from a user's hand. Here, the specific gesture may indicate the user's intention to select a corresponding object. The specific gesture may be predefined.

[0253] According to one embodiment of the present disclosure, the refrigerator (1000) can detect the shape, movement, etc. of a user's hand by analyzing a plurality of image frames. The refrigerator (1000) can detect a specific gesture of the user based on the shape or movement of the user's hand.

[0254] For example, if a specific gesture is a gesture of double-tapping an object with the index finger, the refrigerator (1000) can analyze multiple image frames to detect a gesture of the user double-tapping an object (e.g., a tangerine) with the index finger.

[0255] In step S1830, the refrigerator (1000) according to one embodiment of the present disclosure can identify at least one object selected by the user among objects stored in the refrigerator (1000) by analyzing an image frame acquired at the time when a specific gesture is detected.

[0256] According to one embodiment of the present disclosure, the refrigerator (1000) can identify an object partially overlapping with a user's hand in an image frame acquired at the time a specific gesture is detected as at least one object selected by the user. For example, the refrigerator (1000) can input the image frame acquired at the time a specific gesture is detected into an artificial intelligence model (e.g., an object recognition model), separate and recognize the user's hand and the object in the image frame, and identify the product name of the object or the type of the object. In addition, the refrigerator (1000) can also generate an image of the object by removing the image of the user's hand from the image frame and then applying an inpainting technique to restore a portion of the object that was covered by the user's hand.

[0257] According to one embodiment of the present disclosure, when the refrigerator (1000) identifies at least one object selected by a user, the refrigerator (1000) can add the object to the selected product list (100). At this time, the refrigerator (1000) can also add an image of the object to the selected product list (100). The selected product list (100) can be stored and managed in a storage standby database (e.g., a temporary storage DB). The storage standby database (e.g., a temporary storage DB) can be a DB that stores history only for a certain period of time (e.g., 10 minutes) after the user selects an object.

[0258] In step S1840, the refrigerator (1000) according to one embodiment of the present disclosure may provide information related to at least one object selected by the user.

[0259] According to one embodiment of the present disclosure, information related to at least one object selected by the user may include, but is not limited to, at least one of a storage location of the object, an image of the object, an expiration date of the object, calories of the object, a consumption amount of the object, a degree of spoilage of the object, a degree of discoloration of the object, a price of the object, or an appropriate consumption period of the object.

[0260] According to one embodiment of the present disclosure, the refrigerator (1000) may provide information related to at least one object selected by a user audibly through a speaker or visually through a display unit (1300).

[0261] According to one embodiment of the present disclosure, the refrigerator (1000) can output information related to at least one object through a speaker as a voice when the door (10) is open. When the door (10) of the refrigerator (1000) is open, it may be difficult for the user to check the door display. Therefore, the refrigerator (1000) can provide convenience to the user by providing information related to an object selected by the user through a voice when the door (10) is open. For example, the refrigerator (1000) can output the expiration date of the selected object, the calories of the selected object, the consumption amount of the selected object, the appropriate eating period of the selected object, the degree of discoloration of the selected object, the degree of decay of the selected object, etc. through a voice.

[0262] According to one embodiment of the present disclosure, the refrigerator (1000) may output a notification sound through the speaker of the refrigerator (1000) upon identifying at least one object selected by the user. For example, the refrigerator (1000) may notify the user that the object has been added to the selected product list (100) through a notification sound (or voice).

[0263] According to one embodiment of the present disclosure, when the door (10) is switched from an open state to a closed state, the refrigerator (1000) can output information related to at least one object selected by the user on the display unit (1300). For example, the refrigerator (1000) can output a selected product list (100) including an image of the selected object and a storage location of the selected object on the display unit (1300) of the refrigerator (1000). The selected product list (100) may be provided in the form of a table or may be provided in the form of an internal image of the refrigerator (1000).

[0264] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) may further output at least one of expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, or appropriate eating period information as information related to at least one object selected by the user, on the display unit (1300) of the refrigerator (1000).

[0265] FIG. 19a is a block diagram illustrating the function of a refrigerator (1000) according to one embodiment of the present disclosure. FIG. 19b is a block diagram illustrating the function of a refrigerator (1000) according to one embodiment of the present disclosure.

[0266] As illustrated in FIG. 19a, a refrigerator (1000) according to an embodiment of the present disclosure may include a camera unit (1400), a control unit (1200) (i.e., a processor (1210), a memory (1220)). However, not all of the components illustrated in FIG. 19a are essential components. The refrigerator (1000) may be implemented with more components than the components illustrated in FIG. 19a, or may be implemented with fewer components. For example, as illustrated in FIG. 19b, a refrigerator (1000) according to an embodiment of the present disclosure may include a cooling unit (1100), a control unit (1200), a display unit (1300), a camera unit (1400), a sensor unit (1500), a voice unit (1600), and a communication interface (1700).

[0267] Below, we will look at the above components in turn.

[0268] The cooling unit (1100) may include a compressor (1110), a condenser (1120), an expander (1130), and an evaporator (1140).

[0269] The compressor (1110) compresses the refrigerant to a high temperature and high pressure state. The compressor (1110) can receive electric energy from an external source and compress the gaseous refrigerant to a high temperature and high pressure state using the rotational force of an electric motor or the like. The compressor (1110) is connected to a condenser (1120) and can move the compressed refrigerant to the condenser (1120). The compressor (1110) compresses the refrigerant and pushes it to the condenser (1120), thereby operating the refrigeration cycle of compression, condensation, expansion, and evaporation. Therefore, when the compressor (1110) is in operation, the cold air generated in the evaporator (1140) is supplied to the storage room.

[0270] The condenser (1120) condenses the high-temperature, high-pressure refrigerant compressed from the compressor (1110). The condenser (1120) dissipates heat generated while condensing the refrigerant. The condensed refrigerant passing through the condenser (1120) moves to the expander (1130) (expansion valve). The refrigerant condensed in the condenser (1802) becomes a low-temperature, low-pressure liquid state while passing through the expander (1130) (expansion valve). The liquid refrigerant passes through the expander (1130) (expansion valve) and moves to the evaporator (1140).

[0271] The evaporator (1140) evaporates the low-temperature, low-pressure liquid refrigerant that has passed through the expansion valve. As the liquid refrigerant evaporates, heat exchange occurs with the surrounding gas in the evaporator (1140). As the liquid refrigerant evaporates, it absorbs latent heat from the surroundings, thereby cooling the gas surrounding the evaporator (1140) and generating cold air. The completely evaporated refrigerant is supplied back to the compressor (1110) and the cooling cycle circulates. A heater may be provided around the evaporator (1140) to remove frost formed on the evaporator (1140).

[0272] The control unit (1200) may include a memory (1220) that stores or memorizes a program and / or data for controlling the refrigerator (1000), and a processor (1210) that outputs a control signal for controlling the cooling unit (1100), etc., according to the program and / or data stored in the memory (1220).

[0273] The refrigerator (1000) may include one or more processors (1210). The processor (1210) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a DSP (Digital Signal Processor), and an NPU (Neural Processing Unit). At least one processor (1210) may be implemented in the form of an integrated system on a chip (SoC) including one or more electronic components. Each of the at least one processor (1210) may be implemented as separate hardware (H / W). At least one processor (1210) may be expressed as a MICOM (Micro-Computer, Microprocessor Computer, Microprocessor controller), an MPU (Micro Processor unit), or an MCU (Micro Controller Unit).

[0274] At least one processor (1210) according to the present disclosure may be implemented as a single core processor or as a multicore processor.

[0275] The memory (1220) stores or records various information, data, commands, programs, etc. required for the operation of the refrigerator (1000). The memory (1220) can store temporary data generated during the process of generating a control signal for controlling components included in the refrigerator (1000). The memory (1220) may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0276] The memory (1220) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. The programs stored in the memory (1220) may be classified into a plurality of modules according to their functions.

[0277] The display unit (1300) is for outputting a video signal. The display unit (1300) may include a display panel (1310) and a touch panel (1320). When the display panel (1310) and the touch panel (1320) form a layer structure to form a touch screen, the display unit (1300) may be used as an input device in addition to an output device.

[0278] The display unit (1300) may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In addition, depending on the implementation form of the refrigerator (1000), the refrigerator (1000) may include two or more display units (1300).

[0279] According to one embodiment of the present disclosure, the display unit (1300) may include an LED display (1330). The LED display (1330) may blink in a predetermined color (e.g., red, blue, etc.) when a notification message is output as a voice through the speaker (1620).

[0280] The camera unit (1400) may include an external camera for capturing the external environment and / or an internal camera for capturing the internal environment. There may be multiple internal cameras for capturing the internal environment. For example, the internal cameras may include a main camera (1401) located at the top center of the refrigerator (1000) and sub-cameras (1402) located in each storage compartment.

[0281] The camera unit (1400) may include at least one of an RGB camera, a depth camera, or a thermal imaging camera. In the present disclosure, the camera unit (1400) may be represented by at least one camera.

[0282] The sensor unit (1500) may include at least one of an environmental sensor, a proximity sensor, a door open / close detection sensor, and a spectral sensor, but is not limited thereto. The environmental sensor is a sensor for obtaining environmental information within the refrigerator (1000) and may include at least one of an odor sensor, a temperature sensor, and a humidity sensor. Since the functions of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description thereof will be omitted.

[0283] The voice unit (1600) may include a microphone (1610) and a speaker (1620).

[0284] The microphone (1610) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1610) can receive an acoustic signal (e.g., a voice command) from an external device or a speaker. The microphone (1610) can utilize various noise removal algorithms to remove noise generated during the process of receiving an external acoustic signal.

[0285] The speaker (1620) outputs audio data received from the communication interface (1700) or stored in the memory (1220). In addition, the speaker (1620) outputs an audio signal related to a function performed in the refrigerator (1000) (e.g., a message reception sound, a notification sound).

[0286] The communication interface (1700) may include a short-range communication unit, a long-range communication unit, etc. The short-range wireless communication interface may include a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication interface (NFC), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an IrDA (Infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, an ANT+ communication unit, etc., but is not limited thereto. The long-range communication unit may be used for the refrigerator (1000) to remotely communicate with an external server (2000). The long-range communication unit may include the Internet, a computer network (e.g., LAN or WAN), and a mobile communication unit. The mobile communications unit may include, but is not limited to, 3G modules, 4G modules, 5G modules, LTE modules, NB-IoT modules, LTE-M modules, etc.

[0287] FIG. 20 is a drawing for explaining a communication system of a refrigerator (1000) according to one embodiment of the present disclosure.

[0288] A refrigerator (1000) according to one embodiment of the present disclosure can communicate with an external server (2000), a user terminal (3000), and an external device (4000) through a network.

[0289] The external server (2000) may include a communication module capable of communicating with another server, a refrigerator (1000), an external device (4000), or a user terminal (3000), at least one processor capable of processing data received from another server, a refrigerator (1000), an external device (4000), or a user terminal (3000), and at least one memory capable of storing a program for processing data or processed data. The external server (2000) may be implemented as various computing devices such as a workstation, a cloud, a data drive, or a data station. The external server (2000) may be implemented as one or more servers that are physically or logically separated based on function, detailed configuration of function, or data, and may transmit and receive data and process the transmitted and received data through communication between each server.

[0290] The external server (2000) can perform functions such as managing user accounts, registering a refrigerator (1000) and an external device (4000) by linking them to the user accounts, and managing or controlling the registered refrigerator (1000) and the external device (4000). For example, a user can access the external server (2000) through a user terminal (3000) and create a user account. The user account can be identified by an ID and password set by the user. The external server (2000) can register the refrigerator (1000) and the external device (4000) to the user account according to a set procedure. For example, the external server (2000) can link identification information (e.g., serial number or MAC address) of the refrigerator (1000) and the external device (4000) to the user account, thereby registering, managing, and controlling the refrigerator (1000) and the external device (4000).

[0291] The user terminal (3000) may include a communication module capable of communicating with a refrigerator (1000), an external device (4000), or an external server (2000), a user interface for receiving user input or outputting information to a user, at least one processor for controlling the operation of the user terminal (3000), and at least one memory storing a program for controlling the operation of the user terminal (3000).

[0292] The user terminal (3000) may be carried by the user or placed in the user's home or office, etc. The user terminal (3000) may include, but is not limited to, a personal computer, a terminal, a portable telephone, a smart phone, a handheld device, a wearable device, etc.

[0293] The memory of the user terminal (3000) may store a program, i.e., an application, for controlling the refrigerator (1000) and external devices (4000). The application may be sold installed on the user terminal (3000) or downloaded and installed from an external server.

[0294] By executing an application installed on a user terminal (3000), a user can access an external server (2000) to create a user account, and communicate with the external server (2000) based on the logged-in user account to register a refrigerator (1000) and an external device (4000).

[0295] For example, when the refrigerator (1000) and the external device (4000) are operated so that the refrigerator (1000) and the external device (4000) can be connected to the external server (2000) according to the procedure guided by the application installed on the user terminal (3000), the refrigerator (1000) and the external device (4000) can be registered in the user account by registering the identification information (e.g., serial number or MAC address) of the refrigerator (1000) and the external device (4000) in the corresponding user account on the external server (2000).

[0296] A user can control a refrigerator (1000) and an external device (4000) using an application installed on a user terminal (3000). For example, when a user logs into a user account using an application installed on a user terminal (3000), a refrigerator (1000) and an external device (4000) registered to the user account appear, and when a control command for the refrigerator (1000) or the external device (4000) is input, the control command can be transmitted to the refrigerator (1000) or the external device (4000) via an external server (2000).

[0297] The external device (4000) may include a communication module capable of communicating with a refrigerator (1000), a user terminal (3000), or an external server (2000), a user interface for receiving user input or outputting information to a user, at least one processor for controlling the operation of the external device (4000), and at least one memory storing a program for controlling the operation of the external device (4000).

[0298] The external device (4000) may be at least one of various types of home appliances. For example, the external device (4000) may include, but is not limited to, at least one of a dishwasher, an electric range, an electric oven, an air conditioner, a clothes manager, a washing machine, a dryer, a microwave oven, an air purifier, a robot vacuum cleaner, a vacuum cleaner, and a television.

[0299] A network can include both wired and wireless networks. Wired networks include cable networks or telephone networks, while wireless networks can include any network that transmits and receives signals via radio waves. Wired and wireless networks can be interconnected.

[0300] A network may include a wide area network (WAN) such as the Internet, a local area network (LAN) formed around an access point (AP), and a short-range wireless network that does not use an access point (AP). Short-range wireless networks may include, but are not limited to, Bluetooth™ (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc.

[0301] An access point (AP) can connect a refrigerator (1000), an external device (4000), or a user terminal (3000) to a wide area network (WAN) to which an external server (2000) is connected. The refrigerator (1000), an external device (4000), or a user terminal (3000) can be connected to an external server (2000) via the wide area network (WAN).

[0302] The access point (AP) can communicate with a refrigerator (1000), an external device (4000), or a user terminal (3000) using wireless communication such as Wi-Fi (Wi-Fi™, IEEE 802.11), Bluetooth (Bluetooth™, IEEE 802.15.1), or Zigbee (IEEE 802.15.4), and can connect to a wide area network (WAN) using wired communication, but is not limited thereto.

[0303] According to one embodiment of the present disclosure, the refrigerator (1000) may be directly connected to a user terminal (3000), an external device (1000), or an external server (2000) without going through an access point (AP).

[0304] The refrigerator (1000) may be connected to an external device (4000), a user terminal (3000), or an external server (2000) via a long-range wireless network or a short-range wireless network. For example, the refrigerator (1000) may be connected to a user terminal (3000) via a short-range wireless network (e.g., Wi-Fi Direct).

[0305] The refrigerator (1000) may be connected to a user terminal (3000), an external server (2000), or an external device (4000) via a wide area network (WAN) using a long-distance wireless network (e.g., a cellular communication module). In addition, the refrigerator (1000) may be connected to a wide area network (WAN) using wired communication, and may be connected to a user terminal (3000), an external server (2000), or an external device (4000) via the wide area network (WAN).

[0306] If the refrigerator (1000) can connect to a wide area network (WAN) using wired communication, it may also function as an access relay. Accordingly, the refrigerator (1000) can connect an external device (4000) to a wide area network (WAN) to which an external server (2000) is connected. In addition, the external device (4000) can connect the refrigerator (1000) to a wide area network (WAN) to which an external server (2000) is connected.

[0307] The refrigerator (1000) can transmit information about its operation or status to an external device (4000), a user terminal (3000), or an external server (2000) via a network. For example, the refrigerator (1000) can transmit information about its operation or status to the external device (4000), the user terminal (3000), or the external server (2000) when a request is received from the external server (2000), when a specific event occurs in the refrigerator (1000), or periodically or in real time. When the external server (2000) receives information about its operation or status from the refrigerator (1000), it updates the stored information about the operation or status of the refrigerator (1000), and transmits the updated information about the operation and status of the refrigerator (1000) to the user terminal (3000) via the network. Here, updating information can include various operations that change existing information, such as an operation of adding new information to existing information, or an operation of replacing existing information with new information.

[0308] The refrigerator (1000) can obtain various information from an external device (4000), a user terminal (3000), or an external server (2000), and provide the obtained information to the user. For example, the refrigerator (1000) can obtain information related to the function of the refrigerator (1000) (e.g., recipes, etc.) and various environmental information (e.g., weather, temperature, humidity, etc.) from an external server (2000), and output the obtained information through a user interface.

[0309] The refrigerator (1000) can operate according to a control command received from an external device (4000), a user terminal (3000), or an external server (2000). For example, if the refrigerator (1000) has obtained prior approval from a user so that it can operate according to a control command from an external server (2000) even without a user input, the refrigerator (1000) can operate according to a control command received from the external server (2000). Here, the control command received from the external server (2000) may include, but is not limited to, a control command input by the user through the user terminal (3000) or a control command based on preset conditions.

[0310] The user terminal (3000) can transmit information about the user to the refrigerator (1000), an external device (4000), or an external server (2000) via a communication module. For example, the user terminal (3000) can transmit information about the user's location, health status, preferences, schedule, etc. to the external server (2000). The user terminal (3000) can transmit information about the user to the refrigerator (1000) or the external server (2000) with the user's prior consent.

[0311] The refrigerator (1000), external device (4000), user terminal (3000), or external server (2000) may determine control commands using technologies such as artificial intelligence. For example, the external server (2000) may receive information regarding the operation or status of the refrigerator (1000) and the external device (4000), or receive information regarding the user of the user terminal (3000), process the information using technologies such as artificial intelligence, and transmit the processing result or control command to the refrigerator (1000), the external device (4000), or the user terminal (3000) based on the processing result.

[0312] Below, the operation of the refrigerator (1000) connecting with the user terminal (3000) through the external server (2000) will be examined with reference to FIGS. 21 and 22.

[0313] FIG. 21 is a drawing for explaining an operation in which a refrigerator (1000) according to one embodiment of the present disclosure is connected to a user terminal (3000).

[0314] Referring to FIG. 21, when a selection product list (100) including at least one object selected by a user is generated, the refrigerator (1000) can transmit the selection product list (100) to an external server (2000) via a communication interface (1700). The external server (2000) may be a server that manages a home appliance such as the refrigerator (1000). For example, the refrigerator (1000) can transmit the selection product list (100) to the external server (2000) upon detecting that the door (10) has been switched to a closed state.

[0315] An external server (2000) can transmit a selection product list (100) to a user terminal (3000). At this time, the user terminal (3000) can display a selection product list (100) including at least one object selected by the user from the refrigerator (1000) through an application execution window. For example, when a user selects a selection list item (2101) in the application execution window, the user terminal (3000) can display a selection product list including at least one object selected by the user. With reference to FIG. 22, the operation of the user terminal (3000) displaying the selection product list will be described in more detail.

[0316] FIG. 22 is a diagram for explaining an operation of providing a list of selected products through a user terminal (3000) according to one embodiment of the present disclosure.

[0317] Referring to screen 2210 of FIG. 22, the user terminal (3000) can display a list of selected products (100) using an internal image of the refrigerator (1000). For example, the user terminal (3000) can display a list of selected products (100) by displaying images of multiple objects selected by the user at locations where the objects are stored.

[0318] Referring to screen 2220 of FIG. 22, the user terminal (3000) may display a selection product list (100) in a table format in the application execution window. The selection product list (100) may include images of multiple objects.

[0319] In this case, the user can check the selected product list (100) through the user terminal (3000). If the user needs information about a specific object included in the selected product list (100), the user can select a specific object from the selected product list (100). The user terminal (3000) can provide information related to the specific object in response to a user input for selecting the specific object. For example, if the user selects an egg from the selected product list (100), the user terminal (3000) can provide at least one of the following: expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, appropriate consumption period information, recipe information, and purchase information.

[0320] Additionally, a user may purchase at least one object included in the selected product list (100) via the user terminal (3000). For example, if eggs and carrots are out of stock, the user may add eggs and carrots to the shopping cart from the selected product list (100) and proceed with payment.

[0321] Therefore, according to one embodiment of the present disclosure, a user can check information related to an object selected in the refrigerator (1000) through a user terminal (3000) other than the refrigerator (1000) and conveniently proceed with a purchase.

[0322] According to one embodiment of the present disclosure, a refrigerator (1000) may be provided that tracks the position of a user's hand, identifies an object selected by the user, and provides information related to the object selected by the user.

[0323] A refrigerator (1000) according to one embodiment of the present disclosure may include at least one camera (1400); a memory (1220) for storing storage location information of objects placed in the refrigerator (1000); and at least one processor (1210). The at least one processor (1210) may track the location of a user's hand placed in the refrigerator (1000) using the at least one camera (1400). The at least one processor (1210) may compare the tracked location of the user's hand with the storage location information of objects stored in the memory (1220). Based on the comparison result, the at least one processor (1210) may identify at least one object selected by the user among the objects placed in the refrigerator. The at least one processor (1210) may provide information related to the at least one object selected by the user. According to one embodiment of the present disclosure, the refrigerator (1000) may accurately identify the object selected by the user by tracking the location of the user's hand and comparing the location of the user's hand with the storage locations of the objects.

[0324] According to one embodiment of the present disclosure, at least one processor (1210) can recognize a user's hand in each of image frames acquired through at least one camera (1400). At least one processor (1210) can track the position of the user's hand based on the center point of the user's hand recognized in each of the image frames. According to one embodiment of the present disclosure, the refrigerator (1000) can accurately determine the position of the user's hand by analyzing the image frames.

[0325] At least one processor (1210) according to one embodiment of the present disclosure can identify an object stored at a location that overlaps the location of the user's hand for a predetermined period of time or longer as an object selected by the user.

[0326] At least one processor (1210) according to one embodiment of the present disclosure can output a notification sound through a speaker (1620) of the refrigerator (1000) upon identifying at least one object selected by a user among objects stored in the refrigerator (1000).

[0327] According to one embodiment of the present disclosure, at least one processor (1210) can activate at least one camera (1400) to acquire at least one image frame when the door (10) of the refrigerator (1000) is in an open state. At least one processor (1210) can separately detect a first object and a user's hand from the at least one image frame. At least one processor (1210) can track the positions of each of the first object and the user's hand to determine whether the first object is being stored in the refrigerator (1000) or is being taken out from the refrigerator (1000).

[0328] At least one processor (1210) according to one embodiment of the present disclosure may store an image of the first object, a storage location of the first object, a product name of the first object detected from the image of the first object, and a date of receipt of the first object in a memory (1220) when it is determined that the first object has been received in the refrigerator (1000).

[0329] At least one processor (1210) according to one embodiment of the present disclosure may store the shipping date of the first object in the memory (1220) as it is determined that the first object has been shipped from the refrigerator (1000).

[0330] A refrigerator (1000) according to one embodiment of the present disclosure may include a door open / close detection sensor. At least one processor (1210) may activate at least one camera (1400) when detecting an open state of a door (10) through the door open / close detection sensor. At least one processor (1210) may deactivate at least one camera when detecting a closed state of a door through the door open / close detection sensor.

[0331] At least one camera (1400) according to one embodiment of the present disclosure may include at least one of an RGB camera, a depth camera, or a thermal imaging camera.

[0332] At least one camera (1400) according to one embodiment of the present disclosure may include a main camera (1401) positioned at the top center of the refrigerator (1000) and sub cameras (1402) positioned in each storage compartment.

[0333] At least one processor (1210) according to one embodiment of the present disclosure can output information related to at least one object selected by a user as a voice through a speaker (1620) when the door (10) of the refrigerator (1000) is in an open state.

[0334] According to one embodiment of the present disclosure, at least one processor (1210) can output, as information related to at least one object, an image of at least one object and a storage location of at least one object through the display unit (1300) of the refrigerator (1000), when the door (10) of the refrigerator (1000) is switched from an open state to a closed state.

[0335] At least one processor (1210) according to one embodiment of the present disclosure may output at least one of expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, or appropriate eating period information of at least one object as information related to at least one object to the display unit (1300) of the refrigerator (1000).

[0336] According to one embodiment of the present disclosure, at least one processor (1210) may output a selection product list including at least one object selected by a user to a display unit (1300) of a refrigerator (1000). At least one processor (1210) may receive a user input requesting a purchase of a second object included in the selection product list. In response to the user input, at least one processor (1210) may provide purchase information regarding the second object.

[0337] At least one processor (1210) according to one embodiment of the present disclosure can output recipe information corresponding to a combination of at least one object selected by a user through a display unit (1300) of a refrigerator (1000).

[0338] According to one embodiment of the present disclosure, at least one processor (1210) may detect a user's predetermined gesture with respect to at least one object. In response to the predetermined gesture, the at least one processor (1210) may perform a control command related to at least one object.

[0339] According to one embodiment of the present disclosure, at least one processor (1210) may add at least one object selected by the user to a list of selected products upon detecting a first gesture by the user with respect to at least one object. At least one processor (1210) may provide purchase information for at least one object selected by the user upon detecting a second gesture by the user with respect to at least one object.

[0340] At least one processor (1210) according to one embodiment of the present disclosure may output a notification message as a voice through a speaker (1620) of a refrigerator (1000) if there is a notification message previously output through a user terminal (3000) in relation to at least one object selected by a user.

[0341] At least one processor (1210) according to one embodiment of the present disclosure can blink an LED display (1330) provided inside the refrigerator in a predetermined color when outputting a notification message in voice.

[0342] A method for identifying an object of a refrigerator (1000) according to one embodiment of the present disclosure may include: a step of tracking a position of a user's hand placed in the refrigerator (1000) using at least one camera (1400); a step of comparing the tracked position of the user's hand with storage location information of objects stored in a memory (1220); a step of identifying at least one object selected by the user among objects placed in the refrigerator (1000) based on a result of the comparison; and a step of providing information related to at least one object selected by the user.

[0343] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0344] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM) or a Universal Serial Bus (USB) flash drive), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. At least one camera (1400); A memory (1220) that stores storage location information of objects stored in the refrigerator; and comprising at least one processor (1210), At least one processor, Tracking the position of the user's hand entering the refrigerator (1000) using at least one camera, Compare the location of the user's hand being tracked with the storage location information of objects stored in the memory, Based on the results of the above comparison, at least one object selected by the user among the objects stored in the refrigerator is identified, A refrigerator that provides information related to at least one object selected by the user.

2. In the first paragraph, the at least one processor, Recognizing the user's hand in each of the image frames acquired through the at least one camera, A refrigerator that tracks the position of a user's hand based on the center point of the user's hand recognized in each of the image frames.

3. In the first or second paragraph, the at least one processor, A refrigerator that identifies an object stored in a location that overlaps the location of the user's hand for a predetermined period of time as an object selected by the user.

4. In any one of paragraphs 1 to 3, the at least one processor, A refrigerator that outputs a notification sound through a speaker of the refrigerator when at least one object selected by the user among objects stored in the refrigerator is identified.

5. In any one of paragraphs 1 to 4, the at least one processor, When the door of the refrigerator is open, activating at least one camera to acquire at least one image frame, Separately detecting a first object and the user's hand in at least one image frame, A refrigerator that tracks the positions of each of the first object and the user's hand to determine whether the first object is put into the refrigerator or taken out of the refrigerator.

6. In the fifth paragraph, the at least one processor, As the first object is determined to have been stored in the refrigerator, the image of the first object, the storage location of the first object, the product name of the first object detected from the image of the first object, and the date of storage of the first object are stored in the memory, A refrigerator that stores the shipping date of the first object in the memory as the first object is determined to have been shipped from the refrigerator.

7. In paragraph 5, The above refrigerator further includes a door open / close detection sensor, At least one processor, When the open state of the door is detected through the door open / close detection sensor, at least one camera is activated, A refrigerator that deactivates at least one camera when the door is detected to be closed through the door open / close detection sensor.

8. In any one of paragraphs 1 to 7, the at least one camera, At least one of an RGB camera, a depth camera, or a thermal imaging camera, A refrigerator, wherein the at least one camera comprises a main camera (1401) located at the top center of the refrigerator and sub cameras (1402) located in each storage compartment.

9. In any one of paragraphs 1 to 8, the at least one processor, When the door of the refrigerator is open, information related to at least one object selected by the user is output as a voice through the speaker (1620), A refrigerator that outputs, through the display unit (1300) of the refrigerator, an image of the at least one object and a storage location of the at least one object as information related to the at least one object when the door of the refrigerator is switched from an open state to a closed state.

10. In the 9th paragraph, the at least one processor, A refrigerator further outputting, on a display unit of the refrigerator, at least one of the following information related to the at least one object: expiration date information, calorie information, consumption information, price information, spoilage information, discoloration information, or appropriate consumption period information of the at least one object.

11. In any one of paragraphs 1 to 10, the at least one processor, Outputting a list of selected products including at least one object selected by the user on the display unit of the refrigerator, Receive user input requesting a purchase for a second object included in the above list of selected products; A refrigerator that provides purchase information for the second object in response to the user input.

12. In any one of paragraphs 1 to 11, the at least one processor, A refrigerator that outputs recipe information corresponding to a combination of at least one object selected by the user through a display unit of the refrigerator.

13. In any one of paragraphs 1 to 12, the at least one processor, Detecting a predetermined gesture of the user with respect to at least one object; In response to the above-described gesture, perform a control command related to at least one object, Upon detecting a first gesture of the user for at least one object, adding at least one object selected by the user to a list of selected products, A refrigerator that provides purchase information for at least one object selected by the user upon detecting a second gesture of the user for at least one object.

14. In any one of paragraphs 1 to 13, the at least one processor, If there is a notification message previously output through the user terminal (3000) in relation to at least one object selected by the user, the notification message is output as a voice through the speaker of the refrigerator, A refrigerator that blinks the LED display (1330) provided inside the refrigerator in a predetermined color when the above notification message is output by voice.

15. In a method for identifying an object in a refrigerator, Step (S710) of tracking the position of the user's hand placed in the refrigerator using at least one camera A step (S720) of comparing the location of the user's hand being tracked with the storage location information of objects stored in the memory; Based on the results of the comparison, a step (S730) of identifying at least one object selected by the user among the objects stored in the refrigerator; and A method comprising a step (S740) of providing information related to at least one object selected by the user.

Images