Refrigerator and control method therefor

Abstract

This refrigerator comprises: a main body including a storage compartment; a door connected to the main body; a camera positioned in the main body; and a processor connected to memory and configured to execute one or more instructions stored in the memory. The one or more instructions: control the camera to capture an image of the door and at least a portion of the inside of the main body when opening of the door is detected; identify whether at least one of a first area inside the main body included in the image or a first line included in the door included in the image satisfies a predetermined condition on the basis of the image; and control the refrigerator to store the image as an optimal door bin image when the predetermined condition is satisfied.

Description

Refrigerator and method of controlling the same

[0001] The present disclosure relates to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof capable of obtaining an optimal image for a door bin included in the refrigerator.

[0002] In general, a refrigerator is a home appliance that has a storage compartment for storing food and a cold air supply device that supplies cold air to the storage compartment, allowing the food to be kept fresh for a certain period of time.

[0003] Recently, cameras have been installed in refrigerators to capture images of their interior. Based on the captured images, the refrigerator acquires information about the food currently stored in the refrigerator and provides this information to the user.

[0004] However, in order to obtain information about the food located in the refrigerator door bin, it is necessary to take an optimal image of the door bin.

[0005] According to one embodiment of the present disclosure, a refrigerator comprises: a main body including a storage compartment; a door rotatably coupled to the main body to open and close the storage compartment; a camera positioned on the main body; a memory for capturing images of the inside of the main body and the inside of the door, and storing one or more instructions; and at least one processor operably connected to the memory and configured to execute at least one command stored in the memory; wherein the one or more commands, when executed by the at least one processor, control the camera to capture an image of at least a portion of the inside of the main body and the door when the opening of the door is detected, and identify, based on the image, whether at least one of a first region of the inside of the main body included in the image and a first line included in the door included in the image satisfies a preset condition, and if the preset condition is satisfied, store the image as an optimal door bin image.

[0006] The camera is positioned at the top of the main body, and the first area inside the main body is the area of ​​the bottom and side of the main body covered by the door when the door is closed, and the first line included in the door can be positioned at the top of the door.

[0007] The one or more commands, when executed by the at least one processor, can acquire a template image while the door is open and stopped for a predetermined period of time, compare the template image with an image captured by the camera, and identify whether a first area inside the main body and a first line included in the door satisfy the preset condition based on a result of the comparison between the template image and the image captured by the camera.

[0008] When the one or more commands are executed by the at least one processor, the command detects a change in a first area inside the main body and a first line included in the door included in a plurality of images captured by the camera to determine whether the door is in a moving state, and if the change in the first area inside the main body and the first line included in the door is less than a preset value while the door is open based on the plurality of images, the command determines that the door is open and stopped, and if the command determines that the door is open and stopped for a predetermined period of time, the command may acquire an image acquired at a time when the door is open and stopped as the template image.

[0009] When the one or more commands are executed by the at least one processor, if it is determined that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first preset value, or a length of the first line included in the image captured by the camera is greater than a second preset value, or a door angle acquired based on the first line included in the image captured by the camera is a third preset value, it can be determined that the preset condition is satisfied.

[0010] An output unit; further comprising, when the one or more commands are executed by the at least one processor, when a user input for outputting the optimal doorbin image is received, the output unit can be controlled to output the optimal doorbin image.

[0011] The door includes a first door and a second door, and the one or more commands, when executed by the at least one processor, control the camera to obtain an optimal door bin image corresponding to one of the first and second doors when one of the first and second doors is opened, and control the output unit to output an optimal door bin image corresponding to one of the first and second doors.

[0012] The above camera is a wide-angle camera, and the one or more commands, when executed by the at least one processor, correct an image captured by the wide-angle camera, and identify whether a first area inside the body included in the corrected image and a first line included in the door included in the corrected image satisfy a preset condition.

[0013] An output unit; further comprising, when the one or more commands are executed by the at least one processor, the output unit may be controlled to rotate the optimal doorbin image to a horizontal line based on a transformation guideline, extract and crop a region of interest from the rotated optimal doorbin image to obtain a region of interest image, correct the region of interest image by applying a perspective transform, and output the corrected region of interest image when a user input for outputting the optimal doorbin image is received.

[0014] The one or more commands, when executed by the at least one processor, can control the output unit to obtain information about the groceries whose receipt or shipment has been detected based on the image of the region of interest, and to provide information about the groceries whose receipt or shipment has been detected.

[0015] According to one embodiment of the present disclosure, a method for controlling a refrigerator comprising a main body including a storage compartment, a door including a door bin, a door rotatably coupled to the main body to open and close the storage compartment, and a camera positioned in the main body comprises the steps of: when opening of the door is detected, controlling the camera to capture an image of at least a portion of the interior of the main body and the door; identifying, based on the image, whether at least one of a first area of ​​the interior of the main body included in the image and a first line included in the door included in the image satisfies a preset condition; and, when the preset condition is satisfied, storing the image as an optimal door bin image.

[0016] The camera is positioned at the top of the main body, and the first area inside the main body is the area of ​​the bottom and side of the main body covered by the door when the door is closed, and the first line included in the door can be positioned at the top of the door.

[0017] The step of identifying whether the above-described preset condition is satisfied may include: a step of acquiring a template image while the door is open and stopped for a predetermined period of time; a step of comparing the template image with an image captured by the camera; and a step of identifying whether a first region inside the main body and a first line included in the door satisfy the above-described condition based on the comparison result.

[0018] The step of acquiring the template image may include: a step of detecting a change in a first area inside the main body and a first line included in the door included in a plurality of images captured by the camera; a step of determining whether the door is in a moving state based on the plurality of images; a step of determining that the door is in an open and stationary state when the change in the first area inside the main body and the first line included in the door is less than a preset value while the door is open; and a step of acquiring an image acquired at a time point when the door is in an open and stationary state as the template image when the door is determined to be in an open and stationary state for a predetermined period of time.

[0019] The step of identifying whether the above preset condition is satisfied may be performed when the difference value between the first area included in the template image and the first area included in the image captured by the camera is a first preset value, or the length of the first line included in the image captured by the camera is greater than a second preset value, or the door angle acquired based on the first line included in the image captured by the camera is a third preset value.

[0020] In accordance with one embodiment of the present disclosure, a non-transitory computer-readable medium storing instructions to be executed when executed by a processor of a refrigerator, the refrigerator comprising a main body including a storage compartment, a door bin, a door rotatably coupled to the main body to open and close the storage compartment, and a camera positioned in the main body, wherein a method executed by the processor comprises: when opening of the door is detected, controlling the camera to capture an image of at least a portion of the interior of the main body and the door; identifying, based on the image, whether at least one of a first area of ​​the interior of the main body included in the image and a first line included in the door included in the image satisfies a preset condition; and, if the preset condition is satisfied, storing the image as an optimal door bin image.

[0021] The camera is positioned at the top of the main body, and the first area inside the main body is the area of ​​the bottom and side of the main body covered by the door when the door is closed, and the first line included in the door can be positioned at the top of the door.

[0022] The step of identifying whether the above-described preset condition is satisfied may include: a step of acquiring a template image while the door is open and stopped for a predetermined period of time; a step of comparing the template image with an image captured by the camera; and a step of identifying whether a first region inside the main body and a first line included in the door satisfy the above-described condition based on the comparison result.

[0023] The step of acquiring the template image may include: a step of detecting a change in a first area inside the main body and a first line included in the door included in a plurality of images captured by the camera; a step of determining whether the door is in a moving state based on the plurality of images; a step of determining that the door is in an open and stationary state when the change in the first area inside the main body and the first line included in the door is less than a preset value while the door is open; and a step of acquiring an image acquired at a time point when the door is in an open and stationary state as the template image when the door is determined to be in an open and stationary state for a predetermined period of time.

[0024] The step of identifying whether the above preset condition is satisfied may be performed when the difference value between the first area included in the template image and the first area included in the image captured by the camera is a first preset value, or the length of the first line included in the image captured by the camera is greater than a second preset value, or the door angle acquired based on the first line included in the image captured by the camera is a third preset value.

[0025] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0026] FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to one or more embodiments of the present disclosure;

[0027] FIGS. 2A to 2E are drawings for explaining the internal configuration of a refrigerator according to one or more embodiments of the present disclosure;

[0028] FIG. 3 is a flowchart illustrating a method for a refrigerator to obtain an optimal door bin image according to one or more embodiments of the present disclosure;

[0029] FIG. 4 is a drawing for explaining a first area inside the body and a first line included in the door according to one or more embodiments of the present disclosure.

[0030] FIG. 5A and FIG. 5B are drawings for explaining a method for monitoring door movement of a refrigerator according to one or more embodiments of the present disclosure;

[0031] FIG. 6A and FIG. 6B are drawings for explaining a method for detecting a fully open state of a refrigerator door according to one or more embodiments of the present disclosure;

[0032] FIG. 7 is a diagram illustrating preset conditions for obtaining an optimal doorbin image according to one or more embodiments of the present disclosure;

[0033] FIGS. 8A and 8B are diagrams illustrating optimal door bin images output by a refrigerator according to one or more embodiments of the present disclosure;

[0034] FIGS. 9A to 9C are drawings for explaining an embodiment of obtaining an optimal doorbin image by correcting a photographed image according to one or more embodiments of the present disclosure.

[0035] FIG. 10A and FIG. 10B are diagrams illustrating a method for obtaining an optimal doorbin ease using a learned neural network model according to one or more embodiments of the present disclosure;

[0036] FIGS. 11A to 11D are drawings for explaining a method of correcting a captured optimal doorbin image and providing it to a user according to one or more embodiments of the present disclosure;

[0037] FIGS. 12A to 12C are drawings illustrating a method for providing an optimal doorbin image by removing a human face according to one or more embodiments of the present disclosure;

[0038] FIGS. 13A to 13C are drawings for explaining a method for managing the receipt and delivery of groceries using an optimal door bin image according to one or more embodiments of the present disclosure, and

[0039] FIG. 14 is a flowchart illustrating a method for controlling a refrigerator to obtain an optimal door bin image according to one or more embodiments of the present disclosure.

[0040] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather include various modifications, equivalents, or substitutes of the embodiments.

[0041] In one or more embodiments, similar reference numerals may be used for similar or related components in connection with the description of the drawings.

[0042] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context indicates otherwise.

[0043] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0044] In one or more embodiments, the term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0045] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0046] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0047] In one or more embodiments, the terms “comprises” or “has” and the like are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0048] In one or more embodiments, when a component is said to be “connected,” “coupled,” “supported,” or “contacted” with another component, this includes instances where the components are directly connected, coupled, supported, or contacted, as well as instances where the components are indirectly connected, coupled, supported, or contacted through a third component.

[0049] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0050] A refrigerator according to one or more embodiments may include a body.

[0051] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0052] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0053] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment of the storage room. According to one or more embodiments, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0054] According to one or more embodiments, the insulation may additionally include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0055] A "storage room" may include a space defined by an interior. The storage room may further include an interior defining a space corresponding to the storage room. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0056] A refrigerator may contain one or more storage compartments. When a refrigerator has two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To maintain different temperatures, each compartment may be separated from the others by a partition wall containing insulation.

[0057] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0058] In addition to being called a refrigerator, freezer, and variable temperature room, a storage room can also be called by various other names, such as a vegetable room, a fresh room, a cooling room, and an ice room. The terms refrigerator, freezer, and variable temperature room used below should be understood to encompass storage rooms with corresponding uses and temperature ranges.

[0059] According to one or more embodiments, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close each of one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0060] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0061] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of the doors.

[0062] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door bin for storing items. In this case, the door bin may be referred to as a door basket.

[0063] According to one or more embodiments, the door may include a door body and a front panel that is detachably coupled to a front side of the door body and forms the front of the door. The door body may include a door outer panel that forms the front of the door body, a door inner panel that forms the rear of the door body and faces the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of the doors.

[0064] Refrigerators can be classified into French door types, side-by-side types, bottom-mounted freezers (BMF), top-mounted freezers (TMF), or single-door refrigerators, depending on the arrangement of the doors and storage compartments. French door refrigerators may include a first door and a second door.

[0065] According to one or more embodiments, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0066] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0067] According to one or more embodiments, the cold air supply device can generate cold air through a refrigeration cycle comprising compression, condensation, expansion, and evaporation processes of a refrigerant. To generate cold air, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. According to one or more embodiments, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment through heat generation and cooling through the Peltier effect, whereby an electric current flowing through a junction connecting two materials releases or absorbs heat per unit time at the junction to balance the chemical potential difference between the two materials.

[0068] According to one or more embodiments, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0069] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0070] According to one or more embodiments, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0071] According to one or more embodiments, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0072] According to one or more embodiments, the refrigerator may include a control unit for controlling the refrigerator.

[0073] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0074] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0075] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0076] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0077] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0078] According to one or more embodiments, a refrigerator may include a processor and memory that control all components included in the refrigerator, and may include multiple processors and multiple memories that individually control the components of the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling air supply device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0079] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) to which the refrigerator or user device is connected to the wide area network (WAN) to which the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0080] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0081] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0082] According to one or more embodiments of the present disclosure, an "optimal door bin image" may be an image of a door bin of a door captured while the door is open so as to satisfy preset conditions, and may be an image that satisfies optimal conditions for identifying information about groceries contained in the door bin.

[0083] According to one or more embodiments of the present disclosure, a "template image" may be an image captured with the door fully open to a set angle.

[0084]

[0085] Hereinafter, a refrigerator according to various embodiments will be described in detail with reference to the attached drawings. FIG. 1 is a block diagram illustrating a configuration of a refrigerator according to one or more embodiments of the present disclosure. As illustrated in FIG. 1, a refrigerator (100) may include a camera (110), an output unit (120), a communication interface (130), a microphone (140), a sensor (150), a memory (160), and a processor (170). The refrigerator (100) may be a device for storing food or medicine at a preset temperature to prevent them from being chilled or spoiled. The refrigerator (100) according to one or more embodiments of the present disclosure is illustrated in the shape of a typical household refrigerator, but is not limited thereto, and may be a kimchi refrigerator, a liquor refrigerator, a cosmetics refrigerator, a freezer, etc.

[0086] The camera (110) is configured to capture an image of a subject and generate a captured image, wherein the captured image may include both a moving image and a still image. Meanwhile, the "image" of the present disclosure may be a concept that includes both an image output on a display (123) and an image frame captured by the camera (110). In one or more embodiments, the captured image may be referred to as an image captured by the camera (110).

[0087] As illustrated in FIGS. 1 and 2A, the camera (110) can capture images of the storage compartment inside the main body (230) of the refrigerator (100) and the door bin (or door basket, pantry) area of ​​the doors (210, 220). The camera (110) may be provided in an upper area inside the main body (230) to capture images of at least a portion of the inside of the main body (230) and the door bin area of ​​the doors (210, 220). In addition, the camera (110) may be provided in a side area or a lower area inside the main body (230) to capture images of at least a portion of the inside of the main body (230). In addition, the camera (110) may be provided on the outside of the refrigerator (100) to capture images of the outside of the refrigerator (100). That is, the camera (110) may be implemented not only as a single camera, but may also be implemented as a plurality of cameras depending on the embodiment. This will be described in more detail with reference to FIGS. 2a to 2e.

[0088] FIG. 2A is a drawing illustrating a refrigerator according to one or more embodiments of the present disclosure when the door is open. The refrigerator (100) of FIG. 2A is illustrated as having a plurality of doors (210, 220) on both upper sides, but is not limited thereto, and depending on the arrangement of the doors (210, 220), storage compartments (230), etc., the refrigerator (100) may be implemented as a French door type, a side-by-side type, etc.

[0089] A refrigerator (100) may be provided with doors (210, 220) on both upper sides. In addition, the refrigerator (100) may include a display (123) on at least one of the doors (210, 220) on both upper sides. In this case, not only information about food stored in the refrigerator (100) but also various information (e.g., event information received from the outside, alarm information, recipe information, etc.) may be provided on the display (123).

[0090] The refrigerator (100) may further include a storage compartment (230), as illustrated in FIG. 2A. The storage compartment (230) is accessible by one or more users by opening and closing the doors (210, 220), and may store water, beverages, refrigerated or frozen food. At this time, the storage compartment (230) may include a plurality of storage spaces and storage areas. The storage compartment (230) may be partitioned by partitions arranged within the main body. The storage compartment may be divided into a freezer compartment arranged at the bottom of the refrigerator (100) and a refrigerator compartment arranged at the top. However, the arrangement of the freezer compartment and the refrigerator compartment is not limited thereto, and they may be arranged in an interchangeable position.

[0091] The door (210, 220) can be rotated at an angle set by a hinge (e.g., less than 300°) to open and close a portion of the front of the storage compartment (230).

[0092] At this time, the second door (220) among the plurality of doors (210, 220) may include a display (123) that displays functions and settings of the refrigerator (100) on its surface and can be changed by user input (e.g., selection of a touch or button). In one or more embodiments, at least some of the plurality of doors (210, 220) may further include a dispenser for providing water, ice, or carbonated water and / or a grippable handle.

[0093] In one or more embodiments, as illustrated in FIG. 2A, the refrigerator (100) may include a camera (110) located in the upper region of the main body to capture images of at least a portion of the storage compartment (230) and the door bins of the doors (210, 220). Although FIG. 2A illustrates the camera (110) as being located in the center of the upper region of the main body, this is merely an example. As will be appreciated by those skilled in the art, the camera (110) may be located at any desired location on the main body.

[0094] FIG. 2b is a top perspective view (e.g., top view) of a refrigerator (100), and the camera (110) may be positioned in the upper central area of ​​the main body (e.g., top table) to capture images of both the door bins of the first door (210) and the second door (220), as shown in FIG. 2b. FIG. 2c is a cross-sectional view taken along line A-A' of the drawing of FIG. 2b. The camera (110) may be positioned to face downward at a preset angle (e.g., 30 degrees) to capture images of at least a portion of the storage compartment (230) and the door bins of the doors (210, 220), as shown in FIG. 2c.

[0095] FIG. 2d and FIG. 2e are upper perspective views of the refrigerator (100), illustrating a shooting range (240-1, 240-2) captured by the camera (110). FIG. 2d is a drawing illustrating a shooting range (240-1) captured by the camera (110) when the door (210, 220) of the refrigerator (100) is fully open, and FIG. 2e is a drawing illustrating a shooting range (240-2) captured by the camera (110) when the door (210, 220) of the refrigerator (100) satisfies a preset condition. In one or more embodiments, the preset condition is a condition for obtaining an optimal door bin image, which will be described in more detail later with reference to the drawings.

[0096] In one or more embodiments, the camera (110) may provide the captured image to the processor (170) as an optimal doorbin image.

[0097] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide angle of view, but is not limited thereto. However, the camera (110) may be any suitable camera known to those skilled in the art.

[0098] The output unit or output device (120) can provide various feedbacks. In particular, the output unit or output device (120) may be equipped with a speaker (121), an LED (Light Emitting Diode) (122), a display (123), etc., as illustrated in FIG. 1. However, this is only one embodiment, and other output units (e.g., a haptic providing device, etc.) may be further included.

[0099] At this time, the speaker (121) may be installed inside or outside the refrigerator (100) to provide various auditory feedback through audio. The LED (122) may be installed inside the storage compartment (230) or door (210, 220) inside the refrigerator (100) to provide various visual feedback through indicators of a specific shape (e.g., arrows, etc.) and blinking. The display (123) may be located in at least some areas among the plurality of doors (210, 220) to provide various visual feedback to the user.

[0100] In one or more embodiments, the output unit (120) may output information indicating that groceries have been stored in or taken out of the refrigerator, information about the stored groceries, etc.

[0101] The communication interface (130) can communicate with an external server or an external terminal device. In one or more embodiments, the communication interface (130) can transmit an image including food to an external server to obtain information about the food, and can receive information about the food from the external server. In one or more embodiments, the communication interface (130) can transmit information about the food and information about the storage location of the food to a user terminal, and can receive control commands from the user terminal. In this case, the communication interface (130) can communicate directly with the user terminal, but this is only one embodiment, and it is obvious that the communication interface (130) can communicate with an external user terminal through a server.

[0102] In one or more embodiments, the communication interface (130) may communicate with various external devices using various wireless communication technologies or mobile communication technologies. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared Data Association (IrDA), or near field communication (NFC). Mobile communication technologies may include, for example, 3GPP, Wi-Max, Long Term Evolution (LTE), 5G, and the like.

[0103] In one or more embodiments, the microphone (140) may be installed inside or outside the refrigerator (100) as a component that acquires an audio signal and converts it into an electrical signal. In particular, the microphone (140) may receive an audio signal including a user's voice. In this case, the user's voice may include information about receipt or shipment (hereinafter referred to as "receipt / shipment") and information about food (e.g., type of food, expiration date of food, etc.).

[0104] In one or more embodiments, the sensor (150) may detect an operating state (e.g., power or temperature) of the refrigerator (100) or an external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. In particular, the processor (170) may measure the temperature of each of the plurality of storage compartments of the refrigerator (100) through the sensing values ​​acquired through the sensor (150). In one or more embodiments, the processor (170) may recognize an approach of a user through the sensor (150) and control the camera (110) to be in a ready state. In one or more embodiments, the processor (170) may detect an opening of the door through the sensor (150) and operate the camera (110).

[0105] The memory (160) may store an operating system (OS) for controlling the overall operation of the components of the refrigerator (100) and instructions or data related to the components of the refrigerator (100). In particular, the memory (160) may store various configurations for obtaining an optimal door bin image. In one or more embodiments, the memory (160) may store a database that stores information about food stored in the refrigerator (100) (e.g., type of food, capacity of food, expiration date of food, storage location of food, etc.).

[0106] In one or more embodiments, the memory (160) may store a machine learning model (e.g., a neural network model for object recognition, etc.) for recognizing food items stored in or shipped from the refrigerator (100), according to one or more embodiments. In one or more embodiments, the memory (160) may store a neural network model for obtaining an opening angle of a door (210, 220) included in the refrigerator (100) by inputting an image captured by the camera (110).

[0107] In one or more embodiments, the memory (160) may be implemented as non-volatile memory (e.g., hard disk, solid state drive (SSD), flash memory), volatile memory (which may also include memory within at least one processor (170)), etc.

[0108] The processor (170) can control the refrigerator (100) according to at least one instruction stored in the memory (160).

[0109] In one or more embodiments, the processor (170) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. In one or more embodiments, the one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0110] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an AI-specific processor). For example, an operation for obtaining an optimal door bin image may be performed by a general-purpose processor such as a CPU, and an operation for recognizing a door-frozen angle using a neural network model or an operation for recognizing groceries may be performed by an AI-specific processor such as an NPU.

[0111] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In one or more embodiments, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0112] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0113] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0114] In one or more embodiments, when the opening of the door (210, 220) is detected, the processor (170) controls the camera (110) to capture at least a portion of the interior of the main body and the door (210, 220). The processor (170) identifies (or determines) whether at least one of a first region inside the main body included in the image captured by the camera (110) and a first line included in the door (210, 220) satisfies a preset condition. At this time, the first region inside the main body may be an area of ​​the lower and side of the main body covered by the door (210, 220) when the door (210, 220) is closed, and the first line included in the door (210, 220) may be a line located at the upper end of the door. If it is determined that at least one of the first area inside the main body of the image and the first line included inside the door (210, 220) satisfies a preset condition, the processor (170) stores the image as an optimal doorbin image. At this time, the processor (170) may control the communication interface (130) to not only store the optimal doorbin image but also transmit it to an external device.

[0115] In one or more embodiments, the processor (170) may acquire a template image while the door (210, 220) is fully open (or opened by a preset angle) and stationary for a predetermined period of time. The processor (170) may compare the template image with the captured image to identify whether the first area inside the main body and the first line included in the door satisfy preset conditions.

[0116] In one or more embodiments, the processor (170) may detect a change in a first area inside the main body and a first line included in the door included in a plurality of images captured by the camera (110) to determine whether the door (210, 220) is in a moving state. If the change in the first area inside the main body and the first line included in the door is less than or equal to a preset value while the door (210, 220) is open, the processor (170) may determine that the door (210, 220) is completely open and stationary. If it is determined that the door (210, 220) is completely open and stationary for a predetermined period of time, the processor (170) may acquire an image at the time when the door (210, 220) is completely open and stationary as a template image.

[0117] In one or more embodiments, if it is determined that the difference between the first area included in the template image and the first area included in the captured image is a first preset value, or the length of the first line included in the captured image is greater than a second preset value, or the door angle obtained based on the first line included in the captured image is a third preset value, the processor (170) may determine that the preset condition is satisfied.

[0118] In one or more embodiments, when a user input for outputting an optimal doorbin image is received, the processor (170) can control the output unit (120) to output the optimal doorbin image.

[0119] In one or more embodiments, when one of the first and second doors (210, 220) is opened, the processor (170) may obtain an optimal door bin image corresponding to the opened door. The processor (170) may control the output unit (120) to output the optimal door bin image corresponding to the opened door.

[0120] In one or more embodiments, when the camera (110) is a wide-angle camera, the processor (170) may correct an image captured by the wide-angle camera. The processor (170) may identify whether a first area inside the body and a first line included in the door (210, 220) included in the corrected image satisfy preset conditions.

[0121] In one or more embodiments, the processor (170) may rotate the optimal doorbin image to align it with a horizontal line based on the transformation guideline. The processor (170) may extract and crop a region of interest from the rotated optimal doorbin image to obtain a region of interest image. The processor (170) may correct the region of interest image by applying a perspective transform. When a user input for outputting the optimal doorbin image is received, the processor (170) may control the output unit (120) to output the corrected region of interest image.

[0122] In one or more embodiments, the processor (170) may obtain information about incoming groceries (e.g., groceries stored in a refrigerator) or outgoing groceries (e.g., groceries taken out of a refrigerator) detected through the region of interest image. The processor (170) may control the output unit (120) to provide groceries information including information about the detected incoming or outgoing groceries.

[0123] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0124] FIG. 3 is a flowchart illustrating a method for a refrigerator to obtain an optimal door bin image according to one or more embodiments of the present disclosure.

[0125] First, the refrigerator (100) can detect the opening of the door (S310). At this time, the refrigerator (100) can detect the opening of the door using a sensor provided on at least one of the door (210, 220) and the main body of the refrigerator (100). In one or more embodiments, the refrigerator (100) can accurately detect the open and closed state of the refrigerator door using a Hall sensor located on the lower surface of the upper door and a magnet located on the upper surface of the lower door. In one or more examples, the Hall sensor can be any sensor incorporating one or more Hall elements, each of which can generate a voltage proportional to one axial component of the magnetic field vector B using the Hall effect. The Hall sensor can be used for proximity detection, position confirmation, and speed detection.

[0126] The refrigerator (100) can drive (e.g., control) the camera (110) (S320). At this time, driving the camera (100) is an operation of initiating the operation of the camera (110) to acquire an image, and can be expressed as activating the camera (110) or initiating the camera (110). In one or more embodiments, the above-described embodiment has described driving the camera (110) when detecting the door opening, but this is only one or more embodiments, and it is of course possible to drive the camera (110) when detecting the proximity of a user. When the camera (110) is driven, the refrigerator (100) can acquire an image in real time through the camera (110).

[0127] The refrigerator (100) can monitor door movement (S330). For example, the refrigerator (100) can monitor door movement through changes in images acquired through the camera (110). In particular, the refrigerator (100) can detect changes in a first area inside the main body and a first line included in the door (210, 220) included in multiple images captured through the camera (110) to determine whether the door (210, 220) is moving.

[0128] In one or more embodiments, the first area inside the main body may be at least one area of ​​the bottom and side of the main body that is covered by the door (210, 220) when the door (210, 220) is closed. In one or more embodiments, due to the limitations of the position of the camera (110) and the shooting area, the captured image may show a part of the flange, but not the entire flange, inside the main body (or the storage compartment (230)). For example, the first area inside the main body may include a part of the flange, but not the entire flange, inside the main body (or the storage compartment (230)). In one or more embodiments, the first line included in the door (210, 220) may be a line located at the top of the door (210, 220). At this time, due to the limitations of the position of the camera (110) and the shooting area, the image may show a part of the interior of the door (210, 220), but not the entire interior of the door (210, 220). For example, the “first line included in the door (210, 220)” may be a portion of the length of the upper flange surface of the inner surface constituting the door (210, 220).

[0129] For example, FIG. 3 may be an image captured by a camera (110) of a refrigerator (100) according to one or more embodiments of the present disclosure. At this time, the first region (410) inside the main body may be a portion of the main body located at the bottom of the image, and the first line (420) included in the door (210, 220) may be a portion of the length of the “upper side included in the door” located at the top of the image. Depending on the degree of opening of the refrigerator door, the size of the first region (410) inside the main body and the length or angle of the first line (420) included in the door (210, 220) may change. In one or more embodiments, the degree of opening of the refrigerator door may be determined depending on the size of the first region (410) inside the main body and the length or angle of the first line (420) included in the door (210, 220).

[0130] The refrigerator (100) can determine whether the door is moving by analyzing the images captured by the camera (110). For example, the refrigerator (100) can determine whether the door is moving by analyzing whether the first area has changed from images captured at two arbitrary points in time (t+1, t). In this case, the two arbitrary points in time may be the points in time at which two consecutively captured images were captured.

[0131] In one or more embodiments, the refrigerator (100) can determine whether the door is in a moving state by the following mathematical expression 1.

[0132]

[0133] At this time, A(t+1) may be information about the first region at time t+1, and A(t) may be information about the first region at time t. Here, the information about the first region may be a pixel value of a pixel included in the first region, for example, a value between 0 and 255. Diff may correspond to the absolute value of the difference between A(t+1) and A(t).

[0134] threshold1 and threshold2 may be preset first and second values. In one or more embodiments, the meaning that Diff exceeds the preset first value may mean that the pixel values ​​of pixels included in the first region have changed, and the meaning that the number of pixels for which Diff exceeds the preset first value exceeds the preset second value may mean that a change in pixel values ​​has occurred in a plurality of regions (or a plurality of pixels) within the first region.

[0135] In one or more embodiments, if the size of an area in which the difference in area-specific information (i.e., pixel values) between time points t+1 and t exceeds a preset first value and exceeds a preset second value, the refrigerator (100) may determine that the door is in a moving state. For example, if the number of pixels in which the difference in pixel values ​​between time points t+1 and t exceeds a preset first value and exceeds a preset second value, the refrigerator (100) may determine that the door is in a moving state.

[0136] For example, the refrigerator (100) can acquire an image captured at time t and an image captured at time t+1, as illustrated in FIGS. 5A and 5B . In one or more embodiments, the refrigerator (100) can identify whether the size of an area (520) where the difference between the pixel values ​​included in the first area (510) of the image captured at time t+1 and the pixel values ​​included in the first area (510) of the image captured at time t exceeds a preset first value exceeds a preset second value. If the size of the area (520) where the difference in pixel values ​​exceeds the preset first value exceeds the preset second value, the refrigerator (100) can determine that the door is open.

[0137] In one or more embodiments, the above-described embodiments may determine whether the door is open by a change in the size (or number of pixels) of an area in which pixel values ​​within a first area inside the main body have changed. However, this is only one or more embodiments, and it is also possible to determine whether the door is open by a change in the length of a first line included in the door. In one or more embodiments, the refrigerator may determine whether the door is open by using both a change in the size of the first area and a change in the length of the first line.

[0138] In one or more embodiments, in the above-described embodiments, it is possible to determine whether the door is open by a change in the size (or number of pixels) of an area in which pixel values ​​within a fixed first area inside the main body have changed, but this is only one or more embodiments, and it is possible to determine whether the door is open by a change in the size of the first area. For example, if the first area is not a fixed area but corresponds to a specific area inside the main body of the refrigerator (e.g., a lower area of ​​the main body), it is possible to determine whether the door is open by comparing the sizes of the first areas of images captured at time t and time t+1.

[0139] The refrigerator (100) can identify whether the door (210, 220) is fully open (or opened by a preset angle) and remains stationary for a predetermined period of time (S340). In one or more embodiments, the refrigerator (100) can determine whether the door is fully open and remains stationary for a predetermined period of time by analyzing whether a first region has changed from images captured at two arbitrary points in time (t+1, t). In one or more embodiments, the arbitrary two points in time may be points in time when two consecutively captured images are captured based on camera shooting settings.

[0140] In one or more embodiments, the refrigerator (100) can determine whether the door is in a moving state by the following mathematical expression 2.

[0141]

[0142] In one or more embodiments, A(t+1) may be information about the first region at time t+1, and A(t) may be information about the first region at time t. In one or more embodiments, the information about the first region may be a pixel value of a pixel included in the first region, for example, a value from 0 to 255. Diff may correspond to the absolute value of the difference between A(t+1) and A(t).

[0143] Threshold3 may be a preset third value. In one or more embodiments, the number of pixels where Diff exceeds the preset first value and is less than or equal to the preset third value may mean that there is little change in pixel values ​​within the first region. In one or more embodiments, the preset third value may be the same value as the preset second value described above, but this is only in one or more embodiments and may be a value smaller than the preset second value.

[0144] In one or more embodiments, if the size of the area in which the difference in area information (i.e., pixel values) between time points t+1 and t exceeds a preset first value and is equal to or smaller than a preset third value, the refrigerator (100) may be determined to be fully open (or opened by a preset angle) and stationary for a predetermined period of time. That is, if the number of pixels in which the difference in pixel values ​​between time points t+1 and t exceeds a preset first value and is equal to or smaller than a preset third value, the refrigerator (100) may be determined to be fully open and stationary for a predetermined period of time.

[0145] For example, the refrigerator (100) can acquire an image captured at time t and an image captured at time t+1, as illustrated in FIGS. 6A and 6B. In one or more embodiments, the refrigerator (100) can identify whether the size of an area (620) where the difference between the pixel values ​​included in the first area (610) of the image captured at time t+1 and the pixel values ​​included in the first area (610) of the image captured at time t exceeds the preset first value is equal to or smaller than a preset third value. If the size of the area (620) where the difference in pixel values ​​exceeds the preset first value is equal to or smaller than the preset third value, the refrigerator (100) can determine that it is in a stopped state for a predetermined period of time.

[0146] In particular, when no movement is detected for a certain period of time, the refrigerator (100) can be judged to be in a fully open and stopped state.

[0147] The refrigerator (100) can obtain and store a template image (S350). In one or more embodiments, if it is determined that the door (210, 220) has been in a fully open state for a certain period of time, the refrigerator (100) can obtain an image as a template image at the time point (time point 0) when the door (210, 220) is fully open. In one or more embodiments, the refrigerator (100) can define information about the size of a first area included in the template image as A(t-0). In one or more embodiments, the first area included in the template image can be referred to as a template area, and information about the size of the first area included in the template image can be defined as a template area value.

[0148] For example, FIG. 6a may be a template image obtained by taking a picture at point 0 when the door is fully open.

[0149] In one or more embodiments, the above-described embodiments may determine whether the door is fully open by a change in the size (or number of pixels) of an area in which pixel values ​​within a first area inside the main body have changed, but this is only one embodiment, and it is also possible to determine whether the door is fully open by a change in the length of a first line included in the door. In one or more embodiments, the refrigerator may determine whether the door is fully open by using both a change in the size of an area in which pixel values ​​within the first area have changed and a change in the length of the first line.

[0150] The refrigerator (100) can store a template image, a template area, and a template area value in the memory (160). (In one or more embodiments, the template image, the template area, and the template area value may be collectively referred to as a template.) In one or more embodiments, the refrigerator (100) can obtain an optimal door bin image using at least one of the template image, the template area, and the template area value. In one embodiment, the refrigerator (100) can update and store the template each time the door (210, 220) is opened. Since the environment of the area to be photographed may change each time the door is opened (e.g., ambient lighting, ambient contamination, shadows, etc.), by updating the template each time the door is opened, the accuracy of the optimal door bin image can be improved. For example, by dynamically setting the template each time an optimal door bin image is obtained, a more accurate optimal door bin image can be obtained. In one embodiment, the template may be initially stored in the memory (160) and then used each time the door is opened and closed. For example, after initially obtaining a template, steps S330 to S350 may be omitted. In one embodiment, the template may be updated at a preset interval.

[0151] The refrigerator (100) can identify whether the image captured by the camera (110) satisfies a preset condition (S360). In one or more embodiments, the preset condition is a condition for obtaining an optimal door bin image, and may be a condition related to at least one of the first area and the first line. In one or more embodiments, as illustrated in FIG. 7, the preset condition may be, as illustrated in FIG. 7, 1) an area value (A(t) of the first area (710), 2) a length (l) of the first line (720), 3) an angle ( ) may be a condition for.

[0152] 1) With respect to the condition, when the difference between the size of the first area included in the template image and the size of the first area (710) included in the captured image is a first preset value, the refrigerator (100) can be identified as satisfying the preset condition. For example, the refrigerator (100) can be identified as satisfying the preset condition based on the following mathematical expression 3.

[0153]

[0154] At this time, A(t-0) is information on the first region (or pixel value of the first region) included in the template image, and A(t) may be information (or pixel value) of the first region (710) at any point in time. If the difference between the information on the first region included in the template image and the information on the first region (710) at any point in time satisfies a specific value (S), the refrigerator (100) may be identified as satisfying a preset condition. In one or more embodiments, the specific value may be a value including a preset error. For example, if the size (or number of pixels) of the region where the pixel value within the first region included in the template image and the image captured at any point in time satisfies a specific value (S), the refrigerator (100) may be identified as satisfying a preset condition.

[0155] 2) With respect to the condition, when the length of the first line included in the captured image is greater than a preset value, the refrigerator (100) can be identified as satisfying the preset condition. In one or more embodiments, the refrigerator (100) can be identified as satisfying the preset condition based on the following mathematical expression 4.

[0156]

[0157] In one or more embodiments, the parameter l may be the length of the first line (720) at any point in time. For example, if the length (l) of the first line at any point in time (t) is greater than a specific value, the refrigerator (100) may determine that the preset condition is satisfied.

[0158] 3) With respect to the condition, when the door angle obtained based on the first line (720) included in the captured image is determined to be a third preset value, the refrigerator (100) can be identified as satisfying the preset condition. For example, the refrigerator (100) can be identified as satisfying the preset condition based on the following mathematical expression 5.

[0159]

[0160] At this time, may be the angle between the first line (720) and the reference line (e.g., horizontal line) at any point in time. For example, if the angle between the first line (720) and the reference line (e.g., horizontal line) at any point in time (t) is a specific value, the refrigerator (100) may be determined to satisfy a preset condition. At this time, the angle between the first line (720) and the reference line (e.g., horizontal line) may be determined to be a door angle, and may be an acute angle.

[0161] In one or more embodiments, the refrigerator (100) may determine that the preset condition is satisfied when all of conditions 1), 2), and 3) are satisfied, but this is only one embodiment, and the refrigerator (100) may determine that the preset condition is satisfied when at least one of conditions 1), 2), and 3) is satisfied.

[0162] The refrigerator (100) can acquire an optimal door bin image (S370). Specifically, when a first image satisfying preset conditions is acquired, the refrigerator (100) can acquire the first image as an optimal door bin image and store it in the memory (160). For example, the refrigerator (100) can determine the first image satisfying the preset conditions as an "image at the optimal time for photographing the door bin" and store it in the memory (160) as an optimal door bin image. For example, the refrigerator (100) can store an image as illustrated in FIG. 7 satisfying the preset conditions as an optimal door bin image.

[0163] Meanwhile, in the above-described embodiment, it has been described that an optimal door bin image is obtained for one of the plurality of doors (210, 220), but this is only one embodiment, and an optimal door bin image can be obtained for each of the plurality of doors (210, 220). For example, it goes without saying that the refrigerator (100) can obtain a first optimal door bin image corresponding to the left door (210) and a second optimal door bin image corresponding to the right door (220).

[0164] In one or more embodiments, when a user input for outputting an optimal door bin image is received, the refrigerator (100) may output an optimal door bin image. In one or more embodiments, the user input for outputting the optimal door bin image may be, but is not limited to, a touch input input on a display (123) provided in the refrigerator (100) or a user input input on a user terminal device that is in communication with the refrigerator (100). In another embodiment, when the door of the refrigerator (100) is closed, the refrigerator (100) may output an optimal door bin image through a display (123) provided in the door (210, 220).

[0165] The refrigerator (100) can output at least one of a first optimal door bin image corresponding to the left door (210) and a second optimal door bin image corresponding to the right door (220). For example, as illustrated in FIG. 8A, the refrigerator (100) can output a second optimal door bin image corresponding to the left door (210) (e.g., a left pocket) shown in area 810, and as illustrated in FIG. 8B, the refrigerator (100) can output a first optimal door bin image corresponding to the right door (220) (e.g., a right pocket) shown in area 820. At this time, a UI screen including the first or second optimal door bin image can include information about a door corresponding to each of the first or second optimal door bin images.

[0166] In one or more embodiments, when only one of the plurality of doors (210, 220) included in the refrigerator (100) is opened, the refrigerator (100) may capture only one of the first and second optimal door bin images. For example, when only the left door (210) is opened, the refrigerator (100) may capture only the second optimal door bin image corresponding to the left door (210), and when only the right door (220) is opened, the refrigerator (100) may capture only the first optimal door bin image corresponding to the right door (220). At this time, the refrigerator (100) may output the captured optimal door bin image to the display (123) or transmit it to an external user terminal.

[0167] When the camera (110) is a wide-angle camera for capturing a wide angle of view, as illustrated in FIG. 9A, the corners and edges of the captured image may be distorted. Due to this distortion, a problem may arise in which the accuracy of the operation of acquiring an optimal door bin image may be reduced. In one embodiment, the refrigerator (100) can correct the image captured by the camera (110) (particularly, the wide-angle camera) to identify whether the first area inside the main body and the first line included in the door (210, 220) included in the corrected image satisfy preset conditions.

[0168] For example, when performing the operations of analyzing the image to monitor the movement of the door, acquiring a template image, and acquiring an optimal door bin image in steps S330 to S370 described in FIG. 3, the refrigerator (100) may correct the captured image. In one or more embodiments, as illustrated in FIG. 9A, a distortion phenomenon occurs in which the first region is divided into a plurality of regions (910-1, 910-2) in a curved shape in the image captured by the camera (110), and the first line (920-1, 920-2) becomes a curve, so that the accuracy of determining the optimal door bin image may decrease. Therefore, the refrigerator (100) may perform the aforementioned operations by correcting the captured image, as illustrated in FIGS. 9B and 9C. For example, the refrigerator (100) may correct the captured image based on a pre-stored transformation function. The refrigerator (100) may perform the aforementioned operations based on one first region and one straight line obtained through the image correction. In one embodiment, as illustrated in FIG. 9b, the refrigerator (100) may obtain a template image by determining whether the door (210, 220) is fully open based on the first area (930) of the corrected image. In one or more embodiments, as illustrated in FIG. 9c, the refrigerator (100) may obtain an optimal door bin image by determining whether the first area (940) and the first line (950-1, 950-2) of the corrected image satisfy preset conditions.

[0169] In one or more embodiments, the refrigerator (100) may correct the captured image when outputting an optimal door bin image and output it via the display (123) or transmit it to a user terminal device. In this way, by providing the user with an image with the distortion corrected, aesthetics can be improved and user satisfaction can be enhanced.

[0170] According to one or more embodiments, the refrigerator (100) can recognize the door angle using a learned neural network model. In one or more embodiments, the door angle may correspond to the angle at which the door (210, 220) is opened. The refrigerator (100) can recognize the door angle based on a feature detection model (1020) or an angle detection model (1060). For example, as illustrated in FIG. 10A, the refrigerator (100) can acquire a photographed image (1010). The refrigerator (100) can input the photographed image (1010) into a learned feature detection model (1020) to acquire information about feature points (1030) included in the image (1010). In one or more embodiments, the feature detection model (1020) may be a learned neural network model that inputs an image captured by the refrigerator and extracts feature points of a door included in the image. The refrigerator (100) can input information about the acquired feature point (1030) into the door angle calculation module (1040). At this time, the refrigerator (100) can calculate the door angle based on the position of the feature point (1030) through the door angle calculation module (1040). The refrigerator (100) can obtain information about the door angle of 45 degrees (1050) through the door angle calculation module (1040). As another example, as illustrated in FIG. 10b, the refrigerator (100) can obtain a photographed image (1010). The refrigerator (100) can input the photographed image (1010) into a learned angle detection model (1060) to obtain information about the door angle of the door (210, 220) included in the image (1010). In one or more embodiments, the angle detection model (1060) may be a neural network model trained to input an image captured by the refrigerator and obtain the angle at which the door is opened (i.e., the door angle) contained in the image. The refrigerator (100) may obtain information about the door angle of 45 degrees (1070) through the output of the angle detection model (1060).The refrigerator (100) can obtain an optimal door bin image based on the door angle obtained as described in FIGS. 10a and 10b.

[0171] According to one or more embodiments, the refrigerator (100) may correct the captured optimal door bin image and provide it to the user. That is, the refrigerator (100) may extract a region of interest from the region included in the captured optimal door bin image, and generate a region of interest image based on the region of interest and provide it to the user.

[0172] Specifically, the refrigerator (100) can obtain an optimal door bin image (1110), as illustrated in FIG. 11A. In one or more embodiments, the refrigerator (100) can obtain a plurality of optimal door bin images (i.e., optimal door bin images corresponding to the left door and the right door), but for convenience of explanation, a single optimal door bin image (1110) will be described.

[0173] The refrigerator (100) can rotate the optimal door bin image (1110) to align with the horizontal line based on the conversion guide line (1130), and obtain a rotated optimal door bin image (1120), as illustrated in FIG. 11B. At this time, the conversion guide line (1130) can be a line that guides the door bins of the doors (210, 220) to be positioned in the rotated optimal door bin image, as illustrated in FIG. Specifically, the refrigerator (100) can extract a door bin area in the captured optimal door bin image (1110), and rotate the optimal door bin image (1110) so that the extracted door bin area corresponds to the conversion guide line (1130), thereby obtaining a rotated optimal door bin image (1120).

[0174] The refrigerator (100) can extract and crop a region of interest from a rotated optimal door bin image, as illustrated in FIG. 11C, to obtain a region of interest image (1140). In one or more embodiments, the refrigerator (100) can extract a region of interest (i.e., a region including a door bin region) corresponding to a transformation guideline (1130), crop the extracted region of interest, and enlarge the cropped region of interest to obtain a region of interest image (1140).

[0175] The refrigerator (100) can correct the region of interest image by applying a perspective transform, thereby obtaining a corrected region of interest image (1150), as illustrated in FIG. 11D . In one or more embodiments, the perspective transform may correspond to converting a square into an arbitrary square to express a sense of perspective. In this case, the refrigerator (100) can correct the region of interest image using a pre-stored perspective transform matrix.

[0176] Upon receiving user input for outputting an optimal door bin image, the refrigerator (100) may output a corrected region of interest image (1150), as illustrated in FIG. 11d. The image may be output to the refrigerator's display or transmitted to and displayed on the user's mobile device.

[0177] The optimal door bin image captured by the wide-angle camera makes it difficult to check the groceries depending on the location of the door. However, as described above, by correcting the optimal door bin image to an area of ​​interest image and providing it, the user can more easily check information about the groceries located in the door regardless of the location of the door.

[0178] In the optimal door bin image captured by the refrigerator (100), as illustrated in FIG. 12A, a person (1210) may be captured outside the refrigerator (100) or a person (1220) may be captured near the refrigerator (100) checking the inside of the refrigerator. According to one or more embodiments, the refrigerator (100) may provide an optimal door bin image by removing the unintentionally captured person (1210, 1220). For example, the refrigerator (100) may detect the unintentionally captured person (1210, 1220) and blur or de-identify the detected person (1210, 1220).

[0179] For example, when a person (1210) is captured outside the refrigerator (100), the refrigerator (100) can detect the person (1210) using a face detector, as illustrated in FIG. 12b, and blur the area (1230) where the person is detected to correct the optimal door bin image. As another example, the refrigerator (100) can extract a region of interest (i.e., a door bin area) within the optimal door bin image, and blur areas (1240-1, 1240-2) other than the extracted region of interest, as illustrated in FIG. 12c. In one or more embodiments, when a person (1220) checking the inside of the refrigerator is captured near the refrigerator (100), the refrigerator (100) can blur or de-identify the detected user's face using a face detector. In this case, the de-identification processing may correspond to processing the image so that the person (1220) cannot be identified.

[0180] According to one or more embodiments, the refrigerator (100) may obtain information about groceries whose entry and exit are detected through an image of a region of interest, and may provide groceries information including information about the groceries whose entry and exit are detected. In one or more embodiments, the refrigerator (100) may additionally store shooting information together with the optimal door bin image when shooting an optimal door bin image and storing it in a database. In one or more embodiments, the shooting information may include environmental information when shooting the optimal door bin image, such as the shooting time, information about a user who entered and exited groceries at the time of shooting, the time the user used the refrigerator (the time from when the door was opened to when it was closed), and information about the history of groceries entering and exiting the door bin. For example, when an optimal door bin image (1310) as illustrated in FIG. 13a is stored, the refrigerator (100) can obtain information about "shot at 12:16 PM on November 24, 2023, used by User A (father), closed after 1 minute and 38 seconds of use, milk door taken out, milk storage put in" as shooting information.

[0181] According to one or more embodiments, the refrigerator (100) can recognize groceries stored in and out of the door bin and update a grocery list stored in a database. In one or more embodiments, the refrigerator (100) can obtain an optimal door bin image (1310) as illustrated in FIG. 13A. In addition, the refrigerator (100) can obtain information on a plurality of groceries (1320-1 to 1320-5) contained in the door bin using an object recognition model capable of obtaining information on groceries. In one or more embodiments, the information on groceries may include, but is not limited to, the type of groceries, the number of groceries, the size of groceries, the capacity of groceries, the product name of groceries, etc. Based on the obtained information on groceries (1320-1 to 1320-5), the refrigerator (100) can update a grocery list (1330) as illustrated in FIG. 13C. That is, the refrigerator (100) can add the received groceries to the grocery list (1330) based on the information about the acquired groceries (1320-1 to 1320-5) and delete the shipped groceries from the grocery list (1330). In one or more embodiments, the refrigerator (100) can provide information about the groceries by classifying the grocery list (1330) by storage location. For example, as illustrated in FIG. 13A, when information about the groceries is acquired from the optimal door bin image (1310) corresponding to the right door (220), the refrigerator (100) can update the right door bin grocery list among the grocery list (1330).

[0182] According to one or more embodiments, the refrigerator (100) can detect empty hands or motion of the door (210, 220) to manage the entry and exit details of groceries in the door bin. At this time, the refrigerator (100) can update the grocery list (1330) based on the entry and exit details of groceries in the door bin. Specifically, the refrigerator (100) can detect the entry and exit of groceries in real time through motion detection of a user. At this time, the refrigerator (100) can obtain information on groceries being entered and exited and information on which floor of the door bin the groceries are entered and exited from through motion detection and groceries detection. In addition, the refrigerator (100) can update the grocery list (1330) based on the information on groceries being entered and exited and information on which floor of the door bin the groceries are entered and exited from.

[0183] FIG. 14 is a flowchart illustrating a method for controlling a refrigerator to obtain an optimal door bin image according to one or more embodiments of the present disclosure.

[0184] The refrigerator (100) detects the door opening (S1410). In one or more embodiments, the electronic device (100) may detect the door opening using various sensors, such as a Hall sensor.

[0185] The refrigerator (100) operates the camera (110) to capture images of at least a portion of the interior of the main body and the doors (210, 220) (S1420). In one or more embodiments, the camera (110) may be provided in the upper region of the main body.

[0186] The refrigerator (100) identifies whether at least one of a first area inside the main body and a first line included in the door included in an image captured by the camera satisfies a preset condition (S1430). At this time, the first area inside the main body is an area of ​​the lower and side of the main body covered by the door (210, 220) when the door (210, 220) is closed, and the first line included in the door (210, 220) may be a line located at the upper end of the door (210, 220).

[0187] According to one or more embodiments, the refrigerator (100) can obtain a template image when the door (210, 220) is fully open and stopped for a predetermined period of time. In one or more embodiments, the refrigerator (100) can detect a change in a first area inside the main body and a first line included in the door (210, 220) included in a plurality of images captured by the camera (110) to determine whether the door (210, 220) is in a moving state. If the change in the first area inside the main body and the first line included in the door (210, 220) is less than or equal to a preset value while the door (210, 220) is open, the refrigerator (100) can determine that the door (210, 220) is fully open and stopped. In one or more embodiments, if it is determined that the door (210, 220) is fully open and stopped for a predetermined period of time, the refrigerator (100) can obtain an image at the time when the door (210, 220) is fully open and stopped as a template image.

[0188] And, the refrigerator (100) can identify whether the first area inside the main body and the first line included in the door (210, 220) satisfy the preset conditions by comparing the template image and the photographed image. In one or more embodiments, if the difference value between the size of the first area included in the template image and the first area included in the photographed image is the first preset value, or the length of the first line included in the photographed image is greater than the second preset value, or the door angle obtained based on the first line included in the photographed image is the third preset value, the refrigerator (100) can determine that the preset conditions are satisfied.

[0189] The refrigerator (100) stores a first image that satisfies preset conditions in the first area inside the main body and the first line included inside the door (210, 220) as an optimal door bin image (S1440).

[0190] In one or more embodiments, when a user input for outputting an optimal door bin image is received, the refrigerator (100) can output an optimal door bin image.

[0191] In one or more embodiments, when the door includes a first door and a second door, when one of the first and second doors is opened, the refrigerator (100) can obtain an optimal door bin image corresponding to the opened door. In addition, the refrigerator (100) can output an optimal door bin image corresponding to the opened door.

[0192] In one or more embodiments, the camera (110) is a wide-angle camera, and the refrigerator (100) can correct an image captured by the wide-angle camera. In addition, the refrigerator (100) can identify whether a first area inside the main body and a first line included in the door (210, 220) included in the corrected image satisfy a preset condition.

[0193] In one or more embodiments, the refrigerator (100) may rotate an optimal door bin image to align it with a horizontal line based on a transformation guideline. The refrigerator (100) may extract and crop a region of interest from the rotated optimal door bin image to obtain a region of interest image. The refrigerator (100) may correct the region of interest image by applying a perspective transform. When a user input for outputting an optimal door bin image is received, the refrigerator (100) may output a corrected region of interest image.

[0194] In one or more embodiments, the refrigerator (100) may obtain information about food items detected for receipt or delivery through a region of interest image. The refrigerator (100) may provide food information including information about food items detected for receipt or delivery.

[0195] In one or more embodiments, the processor (170) according to one or more embodiments of the present disclosure controls to process input data according to predefined operating rules or artificial intelligence models stored in the memory (160). The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0196] In one or more embodiments, "learning-based" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data sets. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0197] An artificial intelligence model (e.g., first and second object detection networks) may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0198] A learning algorithm is a method for training a target device using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0199] In one or more embodiments, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0200] In one or more embodiments, the methods according to various embodiments of the present disclosure may be implemented as software including instructions stored on a machine-readable storage medium that can be read by a machine (e.g., a computer). The machine is a device capable of calling the instructions stored from the storage medium and operating according to the called instructions, and may include an electronic device (e.g., a refrigerator) according to the disclosed embodiments.

[0201] In one or more embodiments, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0202] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0203] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the refrigerator, A body including a storage room; A door rotatably coupled to the main body to open and close the storage room; wherein the door includes a door bin; A camera located in the above body; at this time, the camera captures images inside the above body and inside the door, Memory that stores one or more instructions; and At least one processor operably connected to said memory and configured to execute at least one instruction stored in said memory; When said one or more instructions are executed by said at least one processor, When the opening of the door is detected, the camera is controlled to capture an image of at least a portion of the interior of the main body and the door, Based on the image, identify whether at least one of the first area inside the body included in the image and the first line included in the door included in the image satisfies a preset condition; A refrigerator that stores the image as an optimal door bin image if the above preset conditions are satisfied.

2. In paragraph 1, The above camera is located at the top of the main body, The first area inside the main body is the area of ​​the lower part and the side of the main body covered by the door when the door is closed. A refrigerator wherein the first line included in the above door is located at the top of the above door.

3. In paragraph 1, When said one or more instructions are executed by said at least one processor, The template image is acquired while the above door is open and stopped for a certain period of time, Compare the above template image with the image captured by the camera, A refrigerator that identifies whether a first area inside the main body and a first line included in the door satisfy the preset condition based on a comparison result between the template image and the image captured by the camera.

4. In paragraph 3, When said one or more instructions are executed by said at least one processor, Detecting changes in the first area inside the main body and the first line included in the door included in the plurality of images captured by the camera to determine whether the door is in a moving state, If the change in the first area inside the main body and the first line included in the door is less than a preset value while the door is open based on the plurality of images, it is determined that the door is open and stopped. A refrigerator that acquires an image acquired at the time when the door is open and stopped as the template image when it is determined that the door is open and stopped for a certain period of time.

5. In paragraph 3, When said one or more instructions are executed by said at least one processor, A refrigerator that determines that the preset condition is satisfied if a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first preset value, or if the length of the first line included in the image captured by the camera is greater than a second preset value, or if a door angle obtained based on the first line included in the image captured by the camera is a third preset value.

6. In paragraph 1, Including an output section; When said one or more instructions are executed by said at least one processor, A refrigerator that controls the output unit to output the optimal door bin image when a user input for outputting the optimal door bin image is received.

7. In paragraph 6, The above door includes a first door and a second door, When said one or more instructions are executed by said at least one processor, When one of the first and second doors is opened, the camera is controlled to obtain an optimal door bin image corresponding to one of the first and second doors, A refrigerator that controls the output unit to output an optimal door bin image corresponding to one of the first and second doors.

8. In paragraph 1, The above camera is a wide-angle camera, When said one or more instructions are executed by said at least one processor, Corrects the image captured by the above wide-angle camera, A refrigerator that identifies whether a first region inside the main body included in the corrected image and a first line included in the door included in the corrected image satisfy preset conditions.

9. In paragraph 1, Including an output section; When said one or more instructions are executed by said at least one processor, Based on the transformation guidelines, the optimal doorbin image is rotated to align with the horizontal line, Extract and crop the region of interest from the above rotated optimal doorbin image to obtain a region of interest image, Correct the image of the region of interest by applying perspective transform. A refrigerator that controls the output unit to output the corrected region of interest image when a user input for outputting the above optimal door bin image is received.

10. In paragraph 9, When said one or more instructions are executed by said at least one processor, Obtain information about the groceries detected for receipt or shipment based on the image of the region of interest above, A refrigerator that controls the output unit to provide information on the food products detected to be received or shipped.

11. A method for controlling a refrigerator, comprising: a main body including a storage compartment; a door bin; a door rotatably coupled to the main body to open and close the storage compartment; and a camera positioned on the main body. A step of controlling the camera to capture an image of at least a portion of the inside of the main body and the door when the opening of the door is detected; A step of identifying whether at least one of the first area inside the body included in the image and the first line included in the door included in the image satisfies a preset condition based on the image; and A control method comprising: a step of storing the image as an optimal doorbin image if the above preset conditions are satisfied.

12. In paragraph 11, The above camera is located at the top of the main body, The first area inside the main body is the area of ​​the lower part and the side of the main body covered by the door when the door is closed. A control method wherein the first line included in the above door is located at the top of the above door.

13. In paragraph 11, The step of identifying whether the above preset conditions are satisfied is: A step of obtaining a template image while the above door is open and stopped for a certain period of time; A step of comparing the above template image with an image captured by the camera; and A control method comprising: a step of identifying whether a first region inside the main body and a first line included in the door satisfy the preset condition based on the comparison result.

14. In paragraph 13, The step of obtaining the above template image is: A step of detecting a change in a first area inside the main body and a first line included in the door included in a plurality of images captured by the camera; A step of determining whether the door is in a moving state based on the plurality of images; A step of determining that the door is open and stopped when the change in the first area inside the main body and the first line included in the door is less than a preset value while the door is open; and A control method comprising: a step of acquiring an image acquired at a point in time when the door is in an open and stopped state as the template image, if it is determined that the door is in an open and stopped state for a certain period of time.

15. In paragraph 13, The step of identifying whether the above preset conditions are satisfied is: A control method for determining that the preset condition is satisfied when a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first preset value, or when the length of the first line included in the image captured by the camera is greater than a second preset value, or when a door angle acquired based on the first line included in the image captured by the camera is a third preset value.

Images