Refrigerator and control method therefor

Abstract

Provided are a refrigerator and a control method therefor. The refrigerator comprises: a display; a main body including a storage chamber; a door that includes a door bin and is rotatably connected to the main body to open and close the storage chamber; a camera that is located in the main body and captures images of the inside of the main body and the inside of the door; a memory that stores at least one instruction; and a processor which obtains images by imaging at least a portion of the inside of the main body and the door by means of the camera when a trigger signal is detected, detects a food object included in the obtained images, identifies a stocking region of the food object on the basis of the images when it is determined that the food object is being stocked, obtains a cropped image by cropping a region corresponding to the food object in the images, obtains a food image corresponding to the food object on the basis of information about the stocking region of the food object and the cropped image, and adds the obtained food image onto a food UI that visualizes the inside of the refrigerator on the basis of the stocking region of the food object.

Description

Refrigerator and method of controlling the same

[0001] The present disclosure relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof that provides a food UI visualizing the inside of the refrigerator by photographing the inside of the refrigerator.

[0002] In general, a refrigerator is a home appliance that has a storage compartment for storing food and a cold air supply device that supplies cold air to the storage compartment, allowing food to be kept fresh for a long time.

[0003] In particular, recent refrigerators feature cameras embedded in the main body, including the storage compartment, to capture images of the interior. Based on the captured images, the refrigerator acquires information about the food currently stored in the refrigerator and provides this information to the user.

[0004] Meanwhile, conventional technologies have been developed to identify only limited locations of food items or to determine the location of food items using multiple cameras or multiple images. Conventional technologies for determining the location of food items using captured images have limitations, such as not being able to accurately identify food information and locations when multiple food items overlap. Furthermore, technologies for capturing the space inside a refrigerator with a small number of cameras not only have structural limitations, but also suffer from the problem of a significant portion of the refrigerator's interior not being captured due to the limited number and location of cameras. Furthermore, even when combining information about the interior of a refrigerator using multiple cameras or multiple images, there are limitations in identifying or determining food items, such as those that are significantly obscured by other food items or located in blind spots.

[0005] According to one embodiment of the present disclosure, a refrigerator includes: a display; a main body including a storage compartment; a door rotatably coupled to the main body to open and close the storage compartment and including a door bin; a camera positioned in the main body and photographing the inside of the main body and the inside of the door; a memory storing at least one instruction; and a processor configured to, when a trigger signal is detected, photograph at least a portion of the inside of the main body and the door through the camera to obtain an image, detect a grocery object included in the obtained image, identify a storage area of ​​the grocery object based on the image when the grocery object is identified as being stored, crop an area corresponding to the grocery object in the image to obtain a crop image, obtain a grocery image corresponding to the grocery object based on information about the storage area of ​​the grocery object and the crop image, and add the obtained grocery image on a grocery UI that shapes the inside of the refrigerator based on the storage area of ​​the grocery object.

[0006] The above body and door are divided into a plurality of areas, and the storage area of ​​the food object corresponds to one of the plurality of areas, and the processor can add the food image on the storage area of ​​the food object among the plurality of areas included in the food UI.

[0007] The processor can identify a left-right movement direction of the first grocery object based on information about the user's arm angle, the user's arm motion, the user's gaze, and the motion vector obtained from the image, identify a vertical movement direction of the grocery object based on size information of the first grocery object obtained from the image, and identify a storage area of ​​the grocery object based on the left-right movement direction and the vertical movement direction of the first grocery object.

[0008] The processor may mask an area other than an existing grocery image located in a receiving area of ​​the grocery object among a plurality of areas included in the grocery UI, input an image of the masked receiving area and the cropped image into a learned neural network model to obtain a grocery image corresponding to the grocery object, and place the grocery image corresponding to the grocery object on the image of the masked receiving area so as not to overlap with the existing grocery image.

[0009] The processor, when the food object is covered by the user's hand object or another food object, crops the food object covered by the hand object to obtain a cropped image, inputs the obtained cropped image into a learned neural network model to obtain a restored image that restores the portion covered by the hand object, and obtains a food image corresponding to the first food object based on information about the storage area of ​​the first food object and the restored image.

[0010] The processor, when there are multiple food objects, can obtain multiple crop images by cropping each of the multiple food objects, and input each of the multiple crop images into a learned neural network model to obtain multiple restored images in which a portion covered by the hand object is restored.

[0011] The above processor can input the plurality of restored images into a learned neural network model to identify the types of food products included in the plurality of restored images.

[0012] The processor may control the display to generate a first layer corresponding to the grocery image, and to display the first layer and at least one second layer corresponding to an image of a grocery item previously received on the grocery object's receiving area in the grocery UI by overlapping the first layer.

[0013] The processor can control the display to change the display order of the first layer and the at least one second layer according to a user touch.

[0014] The density of groceries displayed on multiple areas included in the above grocery UI may be adjusted according to user settings.

[0015] Meanwhile, according to one embodiment of the present disclosure, a control method of a refrigerator including a main body including a storage compartment, a door rotatably coupled to the main body to open and close the storage compartment and including a door bin, and a camera positioned on the main body and photographing the inside of the main body and the inside of the door, comprises the steps of: when a trigger signal is detected, photographing at least a portion of the inside of the main body and the door through the camera to obtain an image; detecting a food object included in the obtained image; when the food object is identified as being received, identifying a receiving area of ​​the food object based on the image; cropping an area corresponding to the food object in the image to obtain a cropped image; obtaining a food image corresponding to the food object based on information about the receiving area of ​​the food object and the cropped image; and adding the obtained food image to a food UI that shapes the inside of the refrigerator based on the receiving area of ​​the food object.

[0016] The above body and door are divided into a plurality of areas, the storage area of ​​the food object corresponds to one of the plurality of areas, and the adding step may add the food image on the storage area of ​​the food object among the plurality of areas included in the food UI.

[0017] The identifying step may include: identifying a left-right movement direction of the first grocery object based on information about the user's arm angle, the user's arm motion, the user's gaze, and the motion vector obtained from the image; identifying an up-down movement direction of the grocery object based on size information of the first grocery object obtained from the image; and identifying a storage area of ​​the grocery object based on the left-right movement direction and the up-down movement direction of the first grocery object.

[0018] The step of obtaining the grocery image includes: a step of masking an area other than an existing grocery image located in a receiving area of ​​the grocery object among a plurality of areas included in the grocery UI; a step of inputting an image of the masked receiving area and the cropped image into a trained neural network model to obtain a grocery image corresponding to the grocery object; and the step of adding may place the grocery image corresponding to the grocery object on the image of the masked receiving area so as not to overlap with the existing grocery image.

[0019] The step of obtaining the crop image includes a step of obtaining a crop image by cropping the food object covered by the hand object when the food object is covered by the user's hand object or another food object, and a step of inputting the obtained crop image into a learned neural network model to obtain a restored image in which the portion covered by the hand object is restored; and the step of obtaining the food image may obtain a food image corresponding to the first food object based on information about the storage area of ​​the first food object and the restored image.

[0020] The step of obtaining the crop image may include, when there are multiple food objects, a step of obtaining multiple crop images by cropping each of the multiple food objects; and a step of inputting each of the multiple crop images into a learned neural network model to obtain multiple restored images in which a portion covered by the hand object is restored.

[0021] It may include a step of inputting the plurality of restored images into a learned neural network model to identify the type of food included in the plurality of restored images.

[0022] The method may include: generating a first layer corresponding to the grocery image; and displaying the first layer and at least one second layer corresponding to an image of a grocery item previously stored on the grocery object's storage area in the grocery UI by overlapping the first layer.

[0023] The step of displaying the above can be displayed by changing the display order of the first layer and the at least one second layer according to a user touch.

[0024] The density of groceries displayed on multiple areas included in the above grocery UI may be adjusted according to user settings.

[0025] FIG. 1 is a block diagram illustrating the configuration of a refrigerator according to one embodiment of the present disclosure;

[0026] FIGS. 2A to 2F are drawings for explaining the internal and external configuration of a refrigerator according to one embodiment of the present disclosure.

[0027] FIG. 3 is a flowchart illustrating a method for managing the receipt and delivery of groceries according to one embodiment of the present disclosure;

[0028] FIGS. 4A to 4C are drawings for explaining a method of restoring a grocery object according to various embodiments of the present disclosure.

[0029] FIG. 5 is a flowchart illustrating a method for adding received groceries to a grocery UI according to one embodiment of the present disclosure;

[0030] FIG. 6 is a flowchart illustrating a method for identifying a storage area of ​​a grocery object according to one embodiment of the present disclosure;

[0031] FIG. 7 is a diagram illustrating a grocery UI divided into multiple areas according to one embodiment of the present disclosure;

[0032] FIGS. 8 to 10 are drawings for explaining a method for identifying a storage area of ​​a grocery object according to one embodiment of the present disclosure.

[0033] FIG. 11 is a flowchart illustrating a method of generating a grocery image corresponding to a grocery object and adding it to a grocery UI according to one embodiment of the present disclosure;

[0034] FIGS. 12A to 12F are drawings for explaining a method of creating a grocery image corresponding to a grocery object and adding it to a grocery UI according to one embodiment of the present disclosure.

[0035] FIG. 13 is a drawing for explaining a method for providing a grocery UI through multiple layers according to one embodiment of the present disclosure;

[0036] FIG. 14a and FIG. 14b are diagrams for explaining a method of manipulating a grocery UI through a user touch input according to one embodiment of the present disclosure.

[0037] FIG. 15A and FIG. 15B are drawings for explaining a grocery UI having different grocery density according to one embodiment of the present disclosure, and

[0038] FIG. 16 is a drawing for explaining an embodiment of providing some areas among a plurality of areas as a grocery UI according to one embodiment of the present disclosure.

[0039] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or substitutes of the embodiments.

[0040] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0041] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0042] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0043] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0044] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0045] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0046] Terms such as "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0047] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0048] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0049] A refrigerator according to one embodiment may include a body.

[0050] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0051] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0052] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0053] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0054] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0055] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0056] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0057] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0058] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0059] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0060] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0061] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door bin for storing items. The door bin may be referred to as a door basket or door bin.

[0062] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0063] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0064] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0065] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0066] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment by generating heat and cooling through the Peltier effect.

[0067] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0068] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0069] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0070] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0071] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0072] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0073] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0074] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0075] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0076] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0077] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0078] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) to which the refrigerator or user device is connected to the wide area network (WAN) to which the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0079] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0080] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0081] Meanwhile, in the present disclosure, the meaning of the refrigerator (100) “providing” information may include not only displaying information through a display (123) included in the refrigerator (100), but also transmitting information to a user terminal that is in communication with the refrigerator (100) and displaying the information through a display of the user terminal.

[0082] Hereinafter, a refrigerator according to various embodiments will be described in detail with reference to the attached drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to an embodiment of the present disclosure. As illustrated in FIG. 1, a refrigerator (100) may include a camera (110), an output unit (120), a communication interface (130), a microphone (140), a sensor (150), a memory (160), and a processor (170). The refrigerator (100) may be a device for storing food or medicine at a preset temperature to prevent them from being chilled or spoiled. The refrigerator (100) according to an embodiment of the present disclosure is illustrated in the shape of a typical household refrigerator, but is not limited thereto, and may be a kimchi refrigerator, a liquor refrigerator, a cosmetics refrigerator, a freezer, etc.

[0083] The camera (110) is a configuration for capturing a subject to generate a captured image, wherein the captured image may include both a moving image and a still image. Meanwhile, the "image" of the present disclosure may be a concept including both an image output on the display (123) and an image frame captured by the camera (110). In addition, the captured image may include at least one object. According to one embodiment of the present disclosure, the "object" is a configuration included in an image captured by the camera, and may include a food object, a hand object, a face object, a kitchenware object, etc.

[0084] In particular, the camera (110) can capture images of the storage compartment inside the main body (230) of the refrigerator (100) and the door bin (or door basket, pantry) area of ​​the doors (210, 220). The camera (110) can be installed in at least one of the upper area, the lower area, and the side area inside the main body (230) to capture images of the inside of the main body (230) and the door bin area of ​​the doors (210, 220). In addition, the camera (110) can be installed on the outside of the refrigerator (100) to capture images of the outside of the refrigerator (100). That is, the camera (110) can be implemented not only as one camera, but also as a plurality of cameras depending on the embodiment. This will be described in more detail with reference to FIGS. 2A to 2F.

[0085] FIG. 2A is a drawing illustrating a refrigerator according to one embodiment of the present disclosure when the door is open. The refrigerator (100) of FIG. 2A is illustrated as having a plurality of doors (210, 220) on both upper sides, but is not limited thereto, and depending on the arrangement of the doors (210, 220), storage compartments (230), etc., the refrigerator (100) may be implemented as a French door type, a side-by-side type, etc.

[0086] The refrigerator (100) may be provided with doors (210, 220) on both sides of the upper portion. As illustrated in FIG. 2A, the refrigerator (100) may further include a storage compartment (230). The storage compartment (230) is opened by the openable doors (210, 220) and may store water, beverages, refrigerated or frozen food. At this time, the storage compartment (230) may include a plurality of storage spaces and storage spaces. The storage compartment (230) may be divided by partitions arranged inside the main body. The storage compartment may be divided into a freezer compartment arranged at the bottom of the refrigerator (100) and a refrigerator compartment arranged at the top. However, the arrangement of the freezer compartment and the refrigerator compartment is not limited thereto, and they may be arranged in an interchangeable position.

[0087] The door (210, 220) can be rotated at an angle set by a hinge (e.g., less than 300°) to open and close a portion of the front of the storage compartment (230).

[0088] At this time, the second door (220) among the plurality of doors (210, 220) may include a display (123) that displays functions and settings of the refrigerator (100) on its surface and can be changed by user input (e.g., selection of a touch or button). In addition, at least some of the plurality of doors (210, 220) may further include a dispenser that provides water, ice, or carbonated water and / or a grippable handle.

[0089] Meanwhile, as illustrated in FIG. 2a, the refrigerator (100) may include a camera (110) in the upper area of ​​the main body to photograph at least a portion of the storage compartment (230) and the door bin of the door (210, 220).

[0090] FIG. 2b is a top perspective view of a refrigerator (100), and the camera (110) can be positioned in the upper central area of ​​the main body (particularly, the top table) to capture images of both the door bins of the first door (210) and the second door (220), as shown in FIG. 2b. FIG. 2c is a cross-sectional view taken along line A-A' of the drawing of FIG. 2b, and the camera (110) can be positioned to face downward at a preset angle (e.g., 30 degrees) to capture images of at least a portion of the storage compartment (230) and the door bins of the doors (210, 220), as shown in FIG. 2c.

[0091] FIG. 2d and FIG. 2e are upper perspective views of the refrigerator (100), and are drawings illustrating a shooting range (240-1, 240-2) captured by the camera (110). FIG. 2d is a drawing illustrating a shooting range (240-1) captured by the camera (110) when the door (210, 220) of the refrigerator (100) is fully open, and FIG. 2e is a drawing illustrating a shooting range (240-2) captured by the camera (110) when the door (210, 220) of the refrigerator (100) satisfies a preset condition. At this time, the “preset condition” is a condition for obtaining an optimal door bin image, and at this time, the “optimal door bin image” may be an image that satisfies an optimal condition for identifying information on food contained in the door bin.

[0092] In addition, as illustrated in FIG. 2f, the refrigerator (100) may include a display (123) on at least one of the doors (210, 220) on both upper sides. At this time, not only information about food stored in the refrigerator (100) but also various information (e.g., event information received from the outside, alarm information, recipe information, etc.) may be provided on the display (123). At this time, the display (123) may be implemented as a touch screen.

[0093] Additionally, the camera (110) can provide images captured by the processor (170) to manage the receipt and delivery of food.

[0094] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide angle of view, but is not limited thereto.

[0095] Meanwhile, in FIGS. 2A to 2E, the camera (110) is described as being located at the top inside the main body, but this is only one embodiment, and it is obvious that multiple cameras may be provided in other areas inside the main body (e.g., rear area, bottom area, side area, etc.).

[0096] The output unit (120) can provide various feedbacks. In particular, the output unit (120) may be equipped with a speaker (121), an LED (Light Emitting Diode) (122), a display (123), etc., as illustrated in FIG. 1. However, this is only one embodiment, and other output units (e.g., a haptic providing device, etc.) may be further included.

[0097] At this time, the speaker (121) may be installed inside or outside the refrigerator (100) to provide various auditory feedback through audio. The LED (122) may be installed inside the storage compartment (230) or the door (210, 220) inside the refrigerator (100) to provide various visual feedback through indicators of a specific shape (e.g., arrows, etc.) and blinking, etc. of a specific shape. The display (123) may be located in at least some areas among the plurality of doors (210, 220) to provide various visual feedback to the user. In particular, the display (123) may display a grocery UI including information on stocked grocery items. At this time, the grocery UI is a UI that visualizes the inside of the refrigerator (100) and may be divided into a plurality of areas included in the inside of the refrigerator (100). At this time, the plurality of areas may include a plurality of door bin areas installed in the plurality of doors (210, 220) and a plurality of storage compartment areas inside the storage compartment (230). Meanwhile, grocery UI can be called by various terms such as grocery list, grocery information, etc.

[0098] In particular, the output unit (120) can output information indicating that food has been stored in and out of the refrigerator, information on the stored and out food, a food UI, etc. This will be described in detail later with reference to the drawings.

[0099] The communication interface (130) can communicate with an external server or an external terminal device. In particular, the communication interface (130) can transmit an image including food to an external server to obtain information about the food, and can receive information about the food from the external server. In addition, the communication interface (130) can transmit information about the food and information about the storage location of the food to a user terminal, and can receive control commands from the user terminal. In this case, the communication interface (130) can communicate directly with the user terminal, but this is only one embodiment, and it is obvious that the communication interface (130) can communicate with an external user terminal through a server.

[0100] In particular, the communication interface (130) can communicate with various external devices using various wireless communication technologies or mobile communication technologies. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared Data Association (IrDA), or near field communication (NFC), and such mobile communication technologies may include, for example, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, and such.

[0101] The microphone (140) is a component that acquires an audio signal and converts it into an electrical signal, and may be installed inside or outside the refrigerator (100). In particular, the microphone (140) may receive an audio signal including a user's voice. In this case, the user's voice may include information about receipt or shipment (hereinafter referred to as "receipt / shipment") and information about food (e.g., type of food, expiration date of food, etc.).

[0102] The sensor (150) can detect the operating status of the refrigerator (100) (e.g., power or temperature) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. In particular, the processor (170) can measure the temperature of each of the plurality of storage compartments of the refrigerator (100) through the sensing value acquired through the sensor (150). Alternatively, the processor (170) can recognize that a user is approaching through the sensor (150) and control the camera (110) to be in a ready state. Alternatively, the processor (170) can detect the opening of the door through the sensor (150) and drive the camera (110).

[0103] The memory (160) can store an operating system (OS) for controlling the overall operation of the components of the refrigerator (100) and instructions or data related to the components of the refrigerator (100). In particular, the memory (160) can store various configurations for managing the entry and exit of food. In addition, the memory (160) can store a food database (DB) that stores information about food stored in the refrigerator (100) (e.g., type of food, capacity of food, expiration date of food, storage location of food, etc.).

[0104] In addition, the memory (160) may store, according to one embodiment, a learned neural network model (e.g., an object classification model, etc.) for obtaining feature information corresponding to food items stored in or shipped from the refrigerator (100). Alternatively, the memory (160) may store a learned neural network model (e.g., an object recognition model, etc.) for recognizing food items stored in or shipped from the refrigerator (100). Alternatively, the memory (160) may store a learned neural network model (e.g., an object restoration model, etc.) for restoring food items stored in the refrigerator (100). Alternatively, the memory (160) may store a learned neural network model (e.g., an image generation model, etc.) for generating a food image corresponding to food items stored in the refrigerator (100).

[0105] Alternatively, the memory (160) may store an image of a background area photographed from the actual inside of the refrigerator (100). Alternatively, the memory (160) may store a learned neural network model (e.g., a refrigerator image generation model, etc.) for generating a background area inside the refrigerator (100). Meanwhile, the memory (160) may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory), a volatile memory (which may also include a memory within the processor (170)), etc.).

[0106] The processor (170) can control the refrigerator (100) according to at least one instruction stored in the memory (160).

[0107] In particular, the processor (170) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0108] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an AI-dedicated processor). For example, an operation for managing the receipt and delivery of groceries may be performed by a general-purpose processor such as a CPU, and an operation for acquiring feature information corresponding to a groceries object using a neural network model or an operation for recognizing groceries may be performed by an AI-dedicated processor such as an NPU.

[0109] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0110] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0111] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0112] According to one embodiment of the present disclosure, when a trigger signal is detected, the processor (170) captures an image by photographing at least a portion of the inside of the main body and the door through the camera (110). The processor (170) detects a grocery object included in the acquired image. When the grocery object is identified as being received, the processor (170) identifies a receiving area of ​​the grocery object based on the image. The processor (170) crops an area corresponding to the grocery object in the image to obtain a cropped image. The processor (170) obtains a grocery image corresponding to the grocery object based on information about the receiving area of ​​the grocery object and the cropped image. The processor (170) adds the acquired grocery image to a grocery UI that visualizes the inside of the refrigerator based on the receiving area of ​​the grocery object.

[0113] In one embodiment, the main body and the door are divided into multiple areas, and the storage area of ​​the grocery object may correspond to one of the multiple areas. The processor (170) may add a grocery image to the storage area of ​​the grocery object among the multiple areas included in the grocery UI.

[0114] In one embodiment, the processor (170) can identify the left-right movement direction of the first grocery object based on information about the user's arm angle, the user's arm motion, the user's gaze, and the motion vector obtained from the image. The processor (170) can identify the up-down movement direction of the grocery object based on information about the size of the first grocery object obtained from the image. The processor (170) can identify the storage area of ​​the grocery object based on the left-right movement direction and the up-down movement direction of the first grocery object.

[0115] In one embodiment, the processor (170) may mask an area other than an existing grocery image located in a receiving area of ​​a grocery object among a plurality of areas included in a grocery UI. The processor (170) may input an image of the masked receiving area and a cropped image into a trained neural network model to obtain a grocery image corresponding to the grocery object. The processor (170) may place the grocery image corresponding to the grocery object on the image of the masked receiving area so as not to overlap with the existing grocery image.

[0116] In one embodiment, when a grocery object is covered by a user's hand object or another grocery object, the processor (170) may crop the grocery object covered by the hand object to obtain a cropped image. The processor (170) may input the obtained cropped image into a trained neural network model to obtain a restored image that restores the portion covered by the hand object. The processor (170) may obtain a grocery image corresponding to the first grocery object based on information about the storage area of ​​the first grocery object and the restored image.

[0117] In one embodiment, when there are multiple food objects, the processor (170) can obtain multiple cropped images by cropping each of the multiple food objects. The processor (170) can input each of the multiple cropped images into a trained neural network model to obtain multiple restored images that restore the portions covered by the hand object.

[0118] In one embodiment, the processor (170) can input multiple restored images into a learned neural network model to identify the types of food products included in the multiple restored images.

[0119] In one embodiment, the processor (170) may generate a first layer corresponding to a grocery image. The processor (170) may control the display (123) to display the first layer and at least one second layer corresponding to an image of a grocery item previously stored in the grocery object storage area of ​​the grocery UI in an overlapping manner.

[0120] In one embodiment, the processor (170) can control the display (123) to change the display order of the first layer and at least one second layer according to a user touch.

[0121] In one embodiment, the density of groceries displayed on multiple areas included in a grocery UI may be adjustable based on user settings.

[0122] FIG. 3 is a flowchart illustrating a method for managing the receipt and delivery of groceries according to one embodiment of the present disclosure.

[0123] The refrigerator (100) can detect a trigger signal (S305). At this time, the trigger signal is a signal for driving the camera (110), and may be, for example, a trigger signal generated after a door opening is detected by a door open / close sensor, a trigger signal generated after a user's proximity is detected by a proximity sensor, or a trigger signal generated after a user's voice for driving the camera is input through a microphone (140). However, this is merely an example, and the trigger signal may of course be generated by other methods.

[0124] The refrigerator (100) can acquire an image by driving the camera (110) (S310). Driving the camera (110) is an operation that initiates the operation of the camera (110) to acquire an image, and can be expressed as activating the camera (110) or initiating the camera (100). At this time, the refrigerator (100) can acquire an image based on the initial settings of the camera (exposure time, FPS (frames per second), Gain, etc.). In addition, the refrigerator (100) can set a region of interest (ROI) in the image. At this time, the ROI region can be an area where the user's hand or the appearance of groceries being received or delivered can be confirmed.

[0125] In addition, when the camera (110) is located inside, constant lighting and environment can be maintained according to the lighting inside the refrigerator (100), but when the camera (110) is located outside, the shooting environment can change according to the external environment (e.g., external lighting, etc.). Accordingly, the refrigerator (100) can dynamically change the setting values ​​of the camera (110) based on the external environment (e.g., lighting direction, illuminance, etc.).

[0126] The refrigerator (100) can detect a food object (S315). At this time, the food object detected may be a moving food object (e.g., a food object held by a user's hand, a food object contained in a dishware such as a pot, a food object obscured by another food object, etc.). Specifically, the refrigerator (100) can recognize a user's motion using a learned neural network model. At this time, the learned neural network model is a motion recognition model trained to recognize the user's hand motion, and the user's hand motion can be learned by images captured by various external environments. The refrigerator (100) can detect a moving food object and a hand object by recognizing the user's hand motion.

[0127] According to one embodiment, the refrigerator (100) can obtain a cropped image by cropping an area including a food object detected from a photographed image. The refrigerator (100) can input the area including the first food object into a trained first neural network model to obtain first feature information corresponding to the first food object. Alternatively, the refrigerator (100) can input the area including the first food object into a trained second neural network model to obtain information about the first food object.

[0128] Additionally, the refrigerator (100) can detect pantry objects, face objects, kitchenware objects, etc. in addition to hand objects and food objects.

[0129] The refrigerator (100) can track a food object (S320). Specifically, the refrigerator (100) can detect and track the food object for each image frame acquired by the camera (110). At this time, the refrigerator (100) can analyze the movement direction of the food object (or hand object).

[0130] Meanwhile, the refrigerator (100) can acquire information about the food object by inputting the captured image (particularly, a cropped image) into a learned neural network model (e.g., an object recognition model) while detecting and tracking the food object. At this time, the information about the food object may include at least one of the type of the food object, the product name of the food object, the manufacturer of the first food object, and the capacity of the first food object, but is not limited thereto.

[0131] The refrigerator (100) can restore a food object (S325). Specifically, a food object included in an image captured by the camera (110) may be obscured by a user's hand or another food object. The refrigerator (100) can restore a food object obscured by a user's hand or another food object. In this case, "restoration" may refer to an operation of identifying an obscured area among the food objects and filling in the obscured area using the surrounding area of ​​the obscured area. According to one embodiment, the refrigerator (100) can restore a food object obscured by a user's hand or another food object using a learned neural network model (e.g., an object restoration model).

[0132] At this time, the refrigerator (100) may segment the food object from the photographed image, mask the area other than the food object, and input the segmented food object into a trained neural network model to obtain a restored food object in order to restore the food object. In one embodiment, the refrigerator (100) may obtain an image (410) as illustrated in (a) of FIG. 4A. At this time, the image (410) may be a photographed image, but this is merely an example, and may be a cropped image of an area corresponding to the food object included in the photographed image. The refrigerator (100) may segment the food object (420) as illustrated in (b) of FIG. 4A, and mask the area other than the food object (420). In addition, the refrigerator (100) may input the food object (420) into a trained neural network model to obtain a restored food object (430) as illustrated in (c) of FIG. 4A.

[0133] In addition, the refrigerator (100) can restore each of the plurality of objects included in the image. In one embodiment, the refrigerator (100) can obtain an image (440) as illustrated in (a) of FIG. 4b. At this time, the image (440) may include a plurality of food objects of the same type. The refrigerator (100) can segment the first and second food objects (450-1, 450-2) from the image as illustrated in (b) of FIG. 4b, and mask the areas other than the first and second food objects (450-1, 450-2). In addition, the refrigerator (100) can input the first and second food objects (450-1, 450-2) into a trained neural network model to obtain restored first and second food objects (460-1, 460-2) as illustrated in (c) of FIG. By this, by restoring each of the same type of food objects from the captured image, the refrigerator (100) can accurately recognize the number of food objects being received or shipped.

[0134] In another embodiment, the refrigerator (100) may obtain an image (470), as illustrated in (a) of FIG. 4c. At this time, the image (470) may include a plurality of different types of food objects. The refrigerator (100) may segment the first and second food objects (480-1, 480-2) from the image, as illustrated in (b) of FIG. 4c, and mask areas other than the first and second food objects (480-1, 480-2). Then, the refrigerator (100) may input the first and second food objects (480-1, 480-2) into a trained neural network model to obtain restored first and second food objects (490-1, 490-2), as illustrated in (c) of FIG. 4b. By this, by restoring each type of food object from the captured image and inputting an image including the restored multiple types of food objects into an object recognition model, the refrigerator (100) can accurately recognize the type and number of food objects being received or shipped.

[0135] In particular, by restoring a grocery object using a learned neural network model (e.g., an object restoration model) as described above, not only can more accurate feature information be obtained when acquiring feature information using the restored grocery object, but also the restored grocery object can be used to obtain a grocery image to be used in a grocery UI, thereby improving the user experience.

[0136] Meanwhile, in the above-described embodiment, it has been described that one object is restored by inputting an image containing one object into the object restoration model, but this is only one embodiment, and the object restoration model can of course be trained to restore multiple objects simultaneously by inputting an image containing multiple objects.

[0137] The refrigerator (100) can determine whether a food object has been received or shipped (S330). In one embodiment, the refrigerator (100) can determine whether a food object has been received or shipped by analyzing hand and food objects included in a plurality of image frames. In one embodiment, the refrigerator (100) can determine whether a food object has been received or shipped based on the direction of movement of the food object included in the plurality of image frames. That is, if a food object included in a plurality of image frames is identified as moving toward the inside of the main body or toward the door bin, the refrigerator (100) can determine that the food object has been received. Alternatively, if a food object included in a plurality of image frames is identified as moving toward the outside of the refrigerator (100), the refrigerator (100) can determine that the food object has been shipped.

[0138] When it is determined that a food object has been received, the refrigerator (100) can input an image into a trained neural network model (S335). At this time, the neural network model may be a neural network model (e.g., an object classification model) trained to input an image and acquire feature information corresponding to the food object. At this time, the neural network model may be trained by a contrastive learning method. The contrastive learning method is a main learning method of self-supervised learning, and is a method of training such that feature information (or feature values) corresponding to similar images in a vector space are located close to each other, and feature information corresponding to different images are located far apart. In addition, the neural network model can be applied to training by subdividing the label unit, and at this time, the label input by the user can be applied to the training of the neural network model.

[0139] The refrigerator (100) can acquire first feature information from a neural network model (S340). The first feature information may include feature values ​​corresponding to a food object. The first feature information may be used to identify food items to be shipped in the future.

[0140] The refrigerator (100) can match the image with the first feature information (S345). That is, the refrigerator (100) can match and store the image (particularly, the area corresponding to the food object) with the first feature information. In addition, the refrigerator (100) can match and store information about the food object acquired by a neural network model (object recognition model) together with the image and the first feature information. For example, the refrigerator (100) can store information about the food object, such as "the name of the food, the storage location of the food, the date of receipt of the food, the date of shipment, the type of the food, the barcode information of the food, and the text included in the food" together with the image and the first feature information. At this time, the image matching the first feature information may be an image of the area corresponding to the food object, but this is only one embodiment, and may include an image of the food object restored in step S325.

[0141] The refrigerator (100) can update the food DB (S365). Specifically, the refrigerator (100) can match images, first feature information, and information about the food to the food DB and store them in the food DB. At this time, the refrigerator (100) can store the first feature information in a feature information list of the food DB. In particular, the refrigerator (100) can compare the similarity of the newly stored first feature information with other feature information in the feature information list, and sort the first feature information based on the comparison results to update the feature information list. In addition, the feature information list can sort feature information by considering user information such as user patterns, entry / exit times, etc. in addition to the feature information.

[0142] Once the delivery of a food object is determined, the refrigerator (100) can input an image into the trained neural network model (S350). Since the neural network model is identical to the neural network model described in step S335, any duplicate description will be omitted.

[0143] The refrigerator (100) can acquire second feature information from the first neural network model (S355). At this time, the second feature information may include feature values ​​corresponding to food objects. Furthermore, although the second feature information is described as different information from the first feature information to distinguish between incoming and outgoing items, it may have the same value as the first feature information (or a value within a range determined to be the same food type).

[0144] The refrigerator (100) can identify food products that match the second feature information (S360). Specifically, the refrigerator (100) can identify food products that correspond to the feature information that is closest to the second feature information among the feature information stored in the food DB stored in the refrigerator (100). In addition, the refrigerator (100) can identify food products that correspond to the feature information that is closest to the second feature information as food products that match the second feature information. That is, since feature information represents a vector value in a vector space, the refrigerator (100) can obtain information on the similarity between the second feature information and the other feature information by calculating the distance between the second feature information and the other feature information.

[0145] Meanwhile, the refrigerator (100) can obtain food items matching the second feature information and the closest feature information, and can also obtain N candidate feature information items based on the similarity between the second feature information and other feature information, and obtain a candidate list including the N candidate feature information items. In this case, N items may be preset, but this is merely an example and may be changed by a user setting. The refrigerator (100) can receive a user command to select food items shipped to the user using the candidate list.

[0146] The refrigerator (100) can update the food DB (S365). Specifically, the refrigerator (100) can delete food items stored in the food DB that match the second characteristic information. Alternatively, the refrigerator (100) can change information about food items stored in the food DB that match the second characteristic information. For example, the refrigerator (100) can change the number of food items stored in the food DB.

[0147] The refrigerator (100) can provide a grocery UI (or grocery list) using the grocery DB updated in the manner described above. At this time, the grocery UI is a UI that visualizes the interior of the refrigerator (100) and can provide information on grocery items placed in multiple areas within the refrigerator (100). For example, the grocery UI can provide users with information on grocery items stored in multiple areas of the left door bin, information on grocery items stored in multiple areas of the right door bin, information on grocery items stored in the first floor area within the main body, information on grocery items stored in the second floor area within the main body, and information on grocery items stored in the third floor area within the main body. In addition, the grocery UI can include information on grocery items received or shipped. At this time, information on grocery items received or shipped may include information on the time of receipt or shipment, the number of receipts or shipments, etc.

[0148] At this time, the grocery UI may include a grocery object and a background area inside the refrigerator (100). At this time, the background area inside the refrigerator (100) may be a photograph of the inside of an actual refrigerator (100), but this is only an example and may be generated by a learned neural network model.

[0149]

[0150] FIG. 5 is a flowchart illustrating a method for adding stocked groceries to a grocery UI according to one embodiment of the present disclosure. Meanwhile, steps S505, S510, S515, S520, and S525 of FIG. 5 correspond to steps S305, S310, S315, S320, and S330 of FIG. 3 , and therefore, any redundant description will be omitted.

[0151] The refrigerator (100) can recognize the arrival of food (S530). Specifically, the refrigerator (100) can recognize the arrival of food based on the direction of movement of the food object. That is, if the food object moves inward from outside the refrigerator (100), the refrigerator (100) can recognize the arrival of food.

[0152] When the refrigerator (100) is recognized as being stocked, the refrigerator (100) can store a cropped image (S535). That is, the refrigerator (100) can store a cropped image obtained by cropping an area corresponding to a food object from among the captured images. Meanwhile, the cropped image may include the restored image acquired in step S325.

[0153] The refrigerator (100) can obtain information about incoming food items and their storage area (S540). Specifically, the refrigerator (100) can obtain information about incoming food items by inputting captured images into a trained neural network model. Furthermore, the refrigerator (100) can analyze the captured images to identify the direction of movement of the food items, thereby identifying the storage area of ​​the food items.

[0154] A method for identifying a storage area of ​​a food object by analyzing a photographed image of a refrigerator (100) and identifying the direction of movement of the food object will be described with reference to FIGS. 6 to 10.

[0155] FIG. 6 is a flowchart illustrating a method for identifying a storage area of ​​a grocery object according to one embodiment of the present disclosure.

[0156] First, the refrigerator (100) can designate multiple areas within the refrigerator (100) (S610). At this time, the multiple areas can include multiple areas included in the doors (210, 220) and storage compartments (230) within the refrigerator. In particular, the refrigerator (100) can store multiple areas in advance as output values ​​estimated as storage areas of food objects. For example, as illustrated in FIG. 7, the refrigerator (100) can designate first to fourth door bin areas (700-1 to 700-4) to the left door (210), fifth to eighth door bin areas (700-5 to 700-8) to the right door (220), and first to tenth storage compartment areas (710-1 to 710-10) to the storage compartment (230).

[0157] The refrigerator (100) can estimate the left-right direction of entry of a food object (S620). Specifically, the refrigerator (100) can identify the left-right movement direction of the food object being entered based on at least one of information about the user's arm angle, the user's arm motion, the user's gaze, and a motion vector (e.g., optical flow, etc.) obtained from a captured image. In one embodiment, the refrigerator (100) can determine the left-right direction of entry of the food object as one of the following: the direction toward the door bin of the left door (210) as the rightmost direction, the direction toward the left area of ​​the storage compartment (230) as the rightmost direction, the direction toward the right area of ​​the storage compartment (230) as the leftmost direction, and the direction toward the door bin of the left door (210) as the rightmost direction. For example, as illustrated in FIG. 8, when the user's gaze is directed to the left direction (810), the refrigerator (100) can estimate that the entry direction of the food object is the right direction (i.e., the direction toward the left area of ​​the storage compartment (230).

[0158] Meanwhile, in Fig. 8, it is described that the user's gaze direction is estimated based on a single image, but this is only one example, and it is of course possible to estimate the user's gaze direction based on multiple images (i.e., videos).

[0159] The refrigerator (100) can estimate the storage height of the food object (S630). Specifically, the refrigerator (100) can identify the storage height (or vertical movement direction) of the food object based on the size information of the food object obtained from the image. For example, as illustrated in FIG. 9A, if the size of the food object in the first image frame is a first size (910), and as illustrated in FIG. 9B, if the size of the food object in the second image frame, which is the next image frame of the first image frame, is a second size (920) smaller than the first size, the refrigerator (100) can estimate that the storage direction of the food object is the direction in which the food object moves away from the camera (110), i.e., downward. At this time, the refrigerator (100) can estimate the storage height based on the ratio of the first size (910) and the second size (920). Alternatively, the refrigerator (100) can estimate the storage height of the food object by inputting the captured image into a neural network model trained to estimate the storage height of the food object included in the captured image.

[0160] The refrigerator (100) can identify a storage area among a plurality of areas based on the left-right direction of storage and the storage height (S640). That is, the refrigerator (100) can identify a storage area among a plurality of areas by combining the left-right direction of storage and the storage height of a food object. For example, if the left-right storage direction of a food object is to the left and the storage height of the food object is estimated to be the highest, the refrigerator (100) can identify that the second storage area (710-2) is the storage area of ​​the food object, as illustrated in FIG. 10.

[0161] Meanwhile, the refrigerator images illustrated in FIGS. 7 and 10 may be images taken of the inside of an actual refrigerator (100), but this is merely an example and may be virtual refrigerator images generated by a neural network model. In addition, the refrigerator (100) may provide a grocery UI by placing images (actual images or restored images) of grocery objects on the images inside the refrigerator (100).

[0162] In one embodiment, the refrigerator (100) can identify the storage area by considering the motion vector, the angle and motion of the user's arm, the user's line of sight, the size and direction of the object, etc., in order to improve the accuracy of the storage area. Referring again to FIG. 5, the refrigerator (100) can obtain a grocery image corresponding to a grocery object using a learned neural network model (S550). At this time, the neural network model is a neural network model that is learned to obtain a grocery image corresponding to the grocery object by inputting information about the storage area of ​​the grocery object and a crop image (or a restored image), and may be referred to as an image generation model. At this time, the grocery image is an image to be added to the grocery UI, and may be referred to by various terms such as a grocery thumbnail, a grocery UI element, etc.

[0163] The refrigerator (100) can add a grocery image to the grocery UI (S550). Specifically, the refrigerator (100) can add the grocery image to an area of ​​the grocery UI where no existing grocery image is placed. Furthermore, the refrigerator (100) can store the grocery UI with the grocery image added in the memory (160). When a preset event occurs (e.g., an event in which the refrigerator (100) door closes after grocery items are received, or an event in which a user input for displaying the grocery UI is received, etc.), the refrigerator (100) can provide the grocery UI with the grocery image added. Accordingly, the refrigerator (100) can provide the user with the grocery UI with the grocery items added.

[0164] A method for obtaining grocery images and placing them on a grocery UI will be described in more detail with reference to FIGS. 11 to 12f.

[0165] FIG. 11 is a flowchart illustrating a method of generating a grocery image corresponding to a grocery object and adding it to a grocery UI according to one embodiment of the present disclosure.

[0166] The refrigerator (100) can identify the storage area of ​​a food object (S1110). Specifically, the refrigerator (100) can identify the storage area of ​​a food object in the same manner as described in step S540.

[0167] The refrigerator (100) can mask an area other than an existing food image located in the storage area of ​​the food object (S1120). Specifically, the refrigerator (100) can store an image corresponding to the storage area of ​​the food object, as illustrated in FIG. 12A. The image corresponding to the storage area of ​​the food object may include a food image corresponding to a food object stored in the area, along with a background area of ​​the area. Meanwhile, the background area where the food image is placed may be pre-stored, but this is only an example, and may be acquired through a trained neural network model. That is, the refrigerator (100) can input information about the storage area of ​​the food object into the trained neural network model to acquire a background area where the food image is placed.

[0168] The refrigerator (100) can identify an area (1220-1, 1220-2) corresponding to an existing food image from an image (1210) corresponding to a storage area of ​​a food object, as illustrated in FIG. 12b. The refrigerator (100) can obtain a masked image (1230) as illustrated in FIG. 12c by masking an area (e.g., a background area) other than the area (1220-1, 1220-2) corresponding to the existing food image.

[0169] The refrigerator (100) can detect a food object from a cropped image of the food object and perform segmentation (S1130). That is, the refrigerator (100) can obtain a cropped image based on an area corresponding to the food object from an image captured upon receipt. Then, the refrigerator (100) can detect the food object from the cropped image and perform segmentation to obtain the food object. In another embodiment, the refrigerator (100) can detect the food object from the restored image obtained in step S325 of FIG. 3 and perform segmentation to obtain the food object.

[0170] The refrigerator (100) can obtain a food image corresponding to a food object through a learned neural network model (S1140). Specifically, the refrigerator (100) can input information about a storage area (e.g., information about a masked area obtained in step S1120) and a food object on which segmentation has been performed obtained in step S1130 into a learned neural network model (i.e., an image generation model) to obtain a food image corresponding to the stored food object. For example, the refrigerator (100) can input information about a storage area and a food object on which segmentation has been performed into a learned neural network model to obtain a food image (1240) as illustrated in FIG. 12d. Meanwhile, in the above-described embodiment, it has been described that a segmented food object is input into the learned neural network model, but this is merely an example, and it is of course possible to input a cropped image or a restored image into the learned neural network model.

[0171] The refrigerator (100) can place a food image corresponding to a food object on an image of a masked storage area so as not to overlap with existing food images (S1150). For example, as illustrated in FIG. 12e, the refrigerator (100) can place a food image (1250) on an image of a masked storage area so as not to overlap with existing food images (1220-1, 1220-2). At this time, the refrigerator (100) can obtain a background area of ​​the refrigerator (100) for placing the food object. In one embodiment, the refrigerator (100) can obtain a previously stored background area of ​​the refrigerator (100), but this is only one embodiment, and the background area of ​​the refrigerator (100) can be obtained using a learned neural network model. That is, the refrigerator (100) can obtain the background area of ​​the refrigerator (100) by inputting the arrangement information of the food object into the learned neural network model.

[0172] The refrigerator (100) can update and store the grocery UI (S1160). Specifically, the refrigerator (100) can update the image corresponding to the storage area of ​​the grocery object by adding a new grocery image (1250) in addition to the existing grocery images (1220-1, 1220-2). For example, as illustrated in FIG. 12F, the refrigerator (100) can update the image corresponding to the storage area of ​​the grocery object by adding a background area together with the newly added grocery image (1250) in addition to the existing grocery images (1220-1, 1220-2). In addition, the refrigerator (100) can update the grocery UI based on the image corresponding to the storage area of ​​the grocery object and store the updated image in the memory (150).

[0173] As described above, by acquiring an image corresponding to the storage area of ​​the grocery object and updating the grocery UI, it is possible to provide information (i.e., images) about grocery objects currently placed in multiple areas within the refrigerator (100) without a camera for photographing multiple areas within the refrigerator (100).

[0174] Meanwhile, in the above-described embodiment, the grocery images were arranged so as not to overlap with existing grocery images. However, as the number of grocery items increases, overlapping portions may inevitably occur between grocery images. To address this issue, according to one embodiment of the present disclosure, when adding a grocery image, the refrigerator (100) can create a layer corresponding to the grocery image and add the grocery image to the grocery UI.

[0175] Specifically, the refrigerator (100) can generate a first layer corresponding to the grocery image based on the grocery image of the grocery item that has been received. In addition, the refrigerator (100) can superimpose the first layer and at least one second layer corresponding to the previously received grocery item image on the grocery item receiving area of ​​the grocery item in the grocery UI.

[0176] For example, the refrigerator (100) may generate a first layer (1310) corresponding to a first grocery image of the grocery items that have been received, and, as shown on the left side of FIG. 13, may overlap the generated first layer and a plurality of second layers (1320, 1330) corresponding to existing grocery images to provide a grocery UI (i.e., an image corresponding to a receiving area), as shown on the right side of FIG. 13. At this time, the layers may be overlapped in the receiving order of the grocery items included in the grocery items image, but this is merely an example, and may be overlapped based on user preference for the grocery items, frequency of grocery item use, size of the grocery items, expiration date of the grocery items, etc.

[0177] In addition, the refrigerator (100) can change the display order of the first layer and at least one second layer according to a user touch. For example, specifically, as illustrated in FIG. 14a, when the most recently received grocery image (1411) and the existing grocery image (1412) are displayed overlappingly on the grocery UI (1410), the refrigerator (100) can change the display order of the first layer and at least one second layer according to a user touch. For example, as illustrated in FIG. 14a, when a user touch (1420) for dragging in the first direction is input, the refrigerator (100) can change the first layer displayed on the at least one second layer to the back, thereby providing a grocery UI (1410') including existing grocery images (1412, 1413), as illustrated in FIG. 14b. At this time, the most recently received grocery image (1411) may be obscured by existing grocery images (1412, 1413), as illustrated in FIG. 14b. However, this is merely an example, and the most recently received grocery image (1411) may be removed from the grocery UI (1410) based on a user touch.

[0178] In another embodiment, the refrigerator (100) can change the display position of the food image displayed on the food UI according to a user touch. That is, when a user touch is input by touching for a preset period of time and then dragging, the refrigerator (100) can change the display position of the food image according to the drag direction of the user touch.

[0179] As described above, by changing the grocery UI based on the user's touch, a grocery UI including the grocery arrangement desired by the user can be provided. This allows the user to more easily and accurately obtain information about the grocery items placed in the refrigerator (100).

[0180] According to one embodiment of the present disclosure, the density of groceries displayed on a plurality of areas included in a grocery UI may be adjusted according to user settings. For example, when the grocery density is set to a first density, the refrigerator (100) may provide a grocery UI (1510) with the number and arrangement method of grocery images corresponding to the first density, as illustrated in FIG. 15A. When the grocery density is set to a second density, the refrigerator (100) may provide a grocery UI (1520) with the number and arrangement method of grocery images corresponding to the second density, as illustrated in FIG. 15B. In this case, the first density may be higher than the second density, the number of grocery images corresponding to the first density may be greater than or equal to the number of grocery images corresponding to the second density, and the grocery image arrangement method corresponding to the first density may have a larger overlapping area between grocery images than the grocery image arrangement method corresponding to the second density. The density of the refrigerator UI can be set for each of the multiple areas included in the refrigerator UI.

[0181] Meanwhile, according to one embodiment of the present disclosure, not only the number and arrangement of food images, but also the background area in which the food images are arranged can be changed depending on the density. That is, the refrigerator (100) can acquire a background area in which the food images are arranged by inputting information about the storage area of ​​food objects as well as information about the density into a trained neural network model.

[0182] In addition, the refrigerator (100) may provide a food UI (1510, 1520) including information on the entire refrigerator area as illustrated in FIGS. 15A and 15B, but this is merely an example, and as illustrated in FIG. 16, the refrigerator (100) may provide a food UI (1610) including information on at least some areas among a plurality of areas within the refrigerator (100). At this time, at least some areas provided on the food UI (1610) may be selected based on a user input. This allows the user to check in more detail some areas within the refrigerator (100) that he or she wishes to check.

[0183] Meanwhile, the order of the flowcharts of the various embodiments described above is only one example, and the order of each step of the flowchart may be changed, and of course, the order of the flowcharts may be performed simultaneously.

[0184] Meanwhile, according to one embodiment of the present disclosure, the processor (170) controls input data to be processed according to predefined operation rules or artificial intelligence models stored in the memory (160). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0185] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0186] An artificial intelligence model (e.g., first and second object detection networks) may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0187] A learning algorithm is a method for training a target device using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0188] Meanwhile, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0189] The methods according to various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., a refrigerator) according to the disclosed embodiments.

[0190] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0191] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0192] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the refrigerator, display; A body including a storage room; A door rotatably coupled to the main body to open and close the storage room, the door including a door bin; A camera located in the main body and photographing the inside of the main body and the inside of the door; memory for storing at least one instruction; and When a trigger signal is detected, an image is acquired by photographing at least a portion of the interior of the main body and the door through the camera, Detecting food objects included in the acquired image, When the above grocery object is identified as being received, the receiving area of the grocery object is identified based on the image, Obtain a cropped image by cropping the area corresponding to the food object among the above images, Obtaining a grocery image corresponding to the grocery object based on information about the storage area of the grocery object and the crop image; A refrigerator including a processor that adds the acquired grocery image onto a grocery UI that visualizes the inside of the refrigerator based on the storage area of the grocery object.

2. In paragraph 1, The above body and door are divided into multiple areas, The receiving area of the above grocery object corresponds to one of the above multiple areas, The above processor, A refrigerator that adds the grocery image to the receiving area of the grocery object among the multiple areas included in the grocery UI.

3. In paragraph 1, The above processor, Identify the left-right movement direction of the first grocery object based on information about the user's arm angle, the user's arm motion, the user's gaze, and the motion vector obtained from the image, Identifying the up-down movement direction of the first grocery object based on the size information of the first grocery object obtained from the image, A refrigerator that identifies a storage area of a food object based on the left-right movement direction and the up-down movement direction of the first food object.

4. In paragraph 1, The above processor, Among the multiple areas included in the above grocery UI, an area other than the existing grocery image located in the receiving area of the grocery object is masked, By inputting the image of the masked storage area and the cropped image into a learned neural network model, an image of the food corresponding to the food object is obtained. A refrigerator that places a grocery image corresponding to the grocery object on the image of the masked storage area so as not to overlap with the existing grocery image.

5. In paragraph 1, The above processor, If the above grocery object is covered by the user's hand object or another grocery object, crop the grocery object covered by the hand object to obtain a crop image, The above-mentioned acquired crop image is input into a learned neural network model to obtain a restored image that restores the part covered by the hand object. A refrigerator that obtains a grocery image corresponding to the first grocery object based on information about the storage area of the first grocery object and the restored image.

6. In paragraph 5, The above processor, If there are multiple grocery objects, each of the multiple grocery objects is cropped to obtain multiple crop images, A refrigerator that inputs each of the above multiple crop images into a learned neural network model to obtain multiple restored images in which a portion covered by a hand object is restored.

7. In paragraph 6, The above processor, A refrigerator that inputs the plurality of restored images into a learned neural network model to identify the types of groceries included in the plurality of restored images.

8. In paragraph 1, The above processor, Create a first layer corresponding to the above grocery image, A refrigerator that controls the display so as to display, by overlapping the first layer on the receiving area of the grocery object among the grocery UIs, at least one second layer corresponding to an image of a grocery item previously received on the receiving area of the grocery object.

9. In paragraph 8, The above processor, A refrigerator that controls the display to change the display order of the first layer and the at least one second layer according to a user's touch.

10. In paragraph 1, A refrigerator in which the density of groceries displayed on multiple areas included in the above grocery UI can be adjusted according to user settings.

11. A method for controlling a refrigerator, comprising: a main body including a storage compartment; a door including a door bin, the door being rotatably coupled to the main body to open and close the storage compartment; and a camera positioned on the main body and photographing the inside of the main body and the inside of the door. When a trigger signal is detected, a step of capturing an image by photographing at least a part of the inside of the main body and the door through the camera; A step of detecting a food object included in the acquired image; When the above grocery object is identified as being received, a step of identifying a receiving area of the grocery object based on the image; A step of obtaining a cropped image by cropping an area corresponding to the food object among the above images; A step of obtaining a grocery image corresponding to the grocery object based on information about the storage area of the grocery object and the crop image; and A control method comprising: a step of adding the acquired grocery image onto a grocery UI that visualizes the inside of the refrigerator based on the storage area of the grocery object.

12. In paragraph 11, The above body and door are divided into multiple areas, The receiving area of the above grocery object corresponds to one of the above multiple areas, The steps to add above are: A control method for adding a grocery image to a receiving area of a grocery object among a plurality of areas included in the grocery UI.

13. In paragraph 11, The above identifying step is, A step of identifying the left-right movement direction of the first grocery object based on information about the user's arm angle, the user's arm motion, the user's gaze, and the motion vector obtained from the image; A step of identifying the up-down movement direction of the first grocery object based on the size information of the first grocery object obtained from the image; and A control method comprising: a step of identifying a receiving area of the first grocery object based on the left-right movement direction and the up-down movement direction of the first grocery object.

14. In paragraph 11, The steps of obtaining the above grocery image are: A step of masking an area other than an existing grocery image located in the receiving area of the grocery object among a plurality of areas included in the grocery UI; A step of inputting the image of the masked storage area and the cropped image into a learned neural network model to obtain a grocery image corresponding to the grocery object; The steps to add above are: A control method for placing a grocery image corresponding to the grocery object on an image of the masked storage area so as not to overlap with the existing grocery image.

15. In paragraph 11, The steps of obtaining the above crop image are: If the above grocery object is covered by the user's hand object or another grocery object, a step of cropping the grocery object covered by the hand object to obtain a crop image; and A step of inputting the obtained crop image into a learned neural network model to obtain a restored image in which a portion covered by the hand object is restored; The steps of obtaining the above grocery image are: A control method for obtaining a grocery image corresponding to the first grocery object based on information about the storage area of the first grocery object and the restored image.

Images