Refrigerator and control method therefor

Abstract

A refrigerator and a control method therefor are provided. The refrigerator comprises: a camera; at least one memory in which instructions are stored; and at least one processor, wherein the at least one processor individually or collectively executes at least one instruction from among the instructions so as to: control the camera such that an object is captured when the object is put into or withdrawn from the refrigerator; when the captured image of the object put into or withdrawn from the refrigerator is acquired, provide, on the basis of the acquired captured image, a guide for acquiring an additional image; when a signal for capturing the object corresponding to the provided guide is detected, acquire the additional image through the camera; and store the captured image and the additional image as data about the object put into or withdrawn from the refrigerator.

Description

Refrigerator and method of controlling the same

[0001] One or more embodiments of the present disclosure relate to a refrigerator and a control method thereof, and more particularly, to a refrigerator and a control method thereof that manages information about groceries by photographing groceries being stored in and / or shipped from the refrigerator.

[0002] Thanks to advancements in electronic technology, various types of electronic devices are now used in our daily lives. Among these devices are smart refrigerators that utilize neural network models.

[0003] In general, a refrigerator is a home appliance that has a storage compartment for storing food and a cold air supply device that supplies cold air to the storage compartment, allowing food to be kept fresh for a long time.

[0004] In particular, recent refrigerators feature cameras embedded in the main body, including the storage compartment, to capture images of the interior. Based on the captured images, the refrigerator acquires information about the food currently stored in the refrigerator and provides this information to the user.

[0005] According to at least one embodiment of the present disclosure, a refrigerator includes a camera, at least one memory storing instructions, and at least one processor, wherein the at least one processor individually or collectively executes at least one of the instructions to control the camera to photograph an object when the object is received or shipped from the refrigerator, obtain a photographed image of the object received or shipped from the refrigerator, provide a guide for obtaining an additional image based on the obtained photographed image, and when a signal for photographing an object corresponding to the provided guide is detected, obtain an additional image through the camera, and store the photographed image and the additional image as data for the object received or shipped.

[0006] In addition, according to at least one embodiment of the present disclosure, a method for controlling a refrigerator includes, when an object is received or delivered from the refrigerator, a step of photographing the object; when a photographed image of the object received or delivered from the refrigerator is obtained, a step of providing a guide for obtaining an additional image based on the obtained photographed image; when a signal for photographing the object corresponding to the provided guide is detected, a step of obtaining an additional image; and a step of storing the photographed image and the additional image as data for the object received or delivered.

[0007] The above and other aspects, features and advantages of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0008] FIG. 1 is a drawing for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure.

[0009] FIG. 2 is a block diagram illustrating a configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0010] FIG. 3 is a detailed block diagram illustrating a refrigerator according to at least one embodiment of the present disclosure.

[0011] FIG. 4 is a diagram for explaining an operation of obtaining data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0012] FIG. 5 is a diagram for explaining an operation of acquiring data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0013] FIG. 6 is a diagram for explaining an operation of obtaining data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0014] FIG. 7 is a drawing for explaining an operation of acquiring an image of an object being put into or taken out of a refrigerator according to at least one embodiment of the present disclosure.

[0015] FIG. 8 is a drawing for explaining a UI (User Interface) that provides a guide for a refrigerator according to at least one embodiment of the present disclosure.

[0016] FIG. 9 is a drawing for explaining a UI for registering an additional image according to a guide of a refrigerator according to at least one embodiment of the present disclosure.

[0017] FIG. 10 is a diagram illustrating a UI that provides data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0018] FIG. 11 is a diagram illustrating a UI that provides data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0019] FIG. 12 is a drawing for explaining an operation of extracting features for an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0020] FIG. 13 is a diagram illustrating an example of a refrigerator receiving data from various electronic devices according to at least one embodiment of the present disclosure.

[0021] FIG. 14 is a flowchart illustrating an operation for managing data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0022] FIG. 15 is a flowchart illustrating an operation for managing data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0023] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0024] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments.

[0025] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0026] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0027] In this disclosure, when a phrase such as "at least one" precedes a list of items, the phrase modifies the entire list of items and does not modify individual items. Furthermore, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "at least one of A and B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof.

[0028] Terms such as "first," "second," "first," or "second" may be used merely to distinguish one component from another, and do not qualify the components in any other respect (e.g., importance or order).

[0029] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component is connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0030] Terms such as "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0031] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0032] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where another component exists between the component and the other component.

[0033] A refrigerator according to one embodiment may include a body.

[0034] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0035] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0036] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0037] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0038] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0039] A refrigerator may include one or more storage compartments. When a refrigerator has two or more storage compartments, the two or more storage compartments may have different purposes and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0040] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them in a frozen state, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0041] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0042] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0043] The "door" may be configured to seal (or cover) the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0044] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of the door components.

[0045] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door basket for storing items.

[0046] In one embodiment, a door may include a door body and a front panel detachably coupled to a front side of the door body and forming a front surface of the door. The door body may include a door outer panel forming a front surface of the door body, a door inner panel forming a rear surface of the door body and facing a storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of the door components.

[0047] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0048] According to one embodiment, the refrigerator may be configured to include a cold air supply device configured to supply cold air to the storage compartment.

[0049] A "cold air supply device" may include a system comprising a machine, mechanism, electronic device and / or a combination of one or more of these that can generate cold air and guide the cold air to cool a storage room.

[0050] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment by generating heat and cooling through the Peltier effect.

[0051] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0052] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0053] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0054] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0055] According to one embodiment, the refrigerator may be configured to include a control unit for controlling the refrigerator.

[0056] The "control unit" may include at least one memory for storing and / or memorizing a program and / or data related to refrigerator control, and at least one processor for outputting a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0057] At least one memory (hereinafter referred to as "memory") stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. The memory may store temporary data generated during the generation of control signals related to components included in the refrigerator. The memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0058] At least one processor (hereinafter referred to as the "processor") controls the overall operation of the refrigerator. The processor can control components of the refrigerator by executing a program stored in memory. The processor may include a separate NPU that performs the operation of an artificial intelligence model. The processor may also include a central processing unit, a graphics processing unit (GPU), etc. The processor may generate a control signal for controlling the operation of the cold air supply unit. For example, the processor may receive temperature information of a storage compartment from a temperature sensor and generate a cooling control signal for controlling the operation of the cold air supply unit based on the temperature information of the storage compartment.

[0059] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0060] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0061] According to one embodiment, a refrigerator may include at least one processor and at least one memory that control all components included in the refrigerator, and may include multiple processors and multiple memories that individually control the components of the refrigerator. For example, the refrigerator may include at least one processor and at least one memory that control the operation of a cold air supply device based on the output of a temperature sensor. Additionally, the refrigerator may separately include at least one processor and at least one memory that control the operation of a user interface based on user input.

[0062] The communication interface can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) to which the refrigerator and / or user devices are connected to the wide area network (WAN) to which the server is connected. The refrigerator and / or user devices can then connect to the server via the wide area network (WAN).

[0063] For example, the input interface may include, but is not limited to, keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit the user input to the processor.

[0064] For example, the output interface may include, but is not limited to, a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0065] In the present disclosure, the refrigerator “provides” a guide (or guide information) may include not only displaying the guide through a display included in the refrigerator, but also transmitting the guide to a user terminal that is in communication with the refrigerator and displaying the guide through a display of the user terminal.

[0066] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0067] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0068] The various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacings drawn in the attached drawings.

[0069] In this disclosure, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0070] A refrigerator according to an embodiment of the present disclosure will be described in detail with reference to the attached drawings below.

[0071] FIG. 1 is a drawing for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure.

[0072] Referring to FIG. 1, the refrigerator (100) can take pictures of items being received or shipped. For example, when a user places food (10) into the refrigerator (100), the refrigerator can detect the movement of the food (10) and obtain a photographed image of the food (10).

[0073] According to one embodiment, a refrigerator (100) can obtain additional images by providing a guide for obtaining additional images when information regarding object classification is not obtained based on the acquired captured images. The refrigerator (100) can store data about the object based on the captured images and the additional images.

[0074] Here, the term "captured image" may refer to an image obtained by taking a picture of an object. For example, it may refer to an image captured by a refrigerator (100), and may also refer to a captured image obtained from an external device (e.g., a user terminal device, a server device, etc.).

[0075] Here, "additional image" may refer to an image acquired by additionally photographing an object. For example, a photographed image may refer to an image automatically captured of an object being stored or shipped from a refrigerator. An additional image may also refer to an image acquired by additionally photographing an object following a photographing guide after acquiring a photographed image.

[0076] Additional images may include images from various angles sufficient to provide sufficient information about the classification of items being received or shipped from the refrigerator. For example, if the food being received or shipped is identified as a canned beverage, the refrigerator (100) may request additional images of the front, side, and / or side of the canned beverage.

[0077] The "captured image" of the present disclosure may be a concept encompassing additional images, and may also be expressed as a captured video, an image, an additional captured image, a food image, image data, etc.

[0078] In FIG. 1, the refrigerator (100) is illustrated in the shape of a typical household refrigerator, but is not limited thereto, and may be various types of refrigerators, for example, a kimchi refrigerator, a liquor refrigerator, a cosmetics refrigerator, a freezer, etc., but is not limited thereto.

[0079] FIG. 2 is a block diagram illustrating a configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0080] Referring to FIG. 2, a refrigerator (100) may include a camera (110), one or more memories (120, hereinafter referred to as “memory”), and one or more processors (130). The refrigerator (100) may be a device for storing food or medicine at a preset temperature or a lower temperature to prevent them from spoiling or cooling.

[0081] Items that are stored in or shipped from the refrigerator (100) (hereinafter, simply referred to as “storage”) include items that require storage at a preset temperature or a lower temperature, such as food and cosmetics. For convenience of explanation, the following description will mainly focus on food as an example.

[0082] The camera (110) can capture an object and generate a captured image, wherein the captured image may include both a moving image and a still image. The term "image" in the present disclosure may be a concept that includes both an image output on a display and an image frame captured by the camera (110).

[0083] In particular, the camera (110) can capture images of the storage compartment inside the main body of the refrigerator (100) and the door bin (or door basket, pantry, etc.) area of ​​the door. The camera (110) can be installed in at least one of the upper area, the lower area, and the side area inside the main body to capture images of the inside of the main body and the door bin area of ​​the door. In addition, the camera (110) can be installed on the outside of the refrigerator (100) to capture images of the outside of the refrigerator (100). That is, the camera (110) can be implemented not only as a single camera, but also as a plurality of cameras depending on the embodiment.

[0084] The refrigerator (100) may include a camera (110) in the upper region of the main body to capture images of at least a portion of the storage compartment and the door bin of the door. However, the present invention is not limited thereto, and the camera (110) may be located in another region (e.g., a rear region, a bottom region, a side region, etc.) inside the main body, or may be provided in multiple numbers.

[0085] Additionally, the camera (110) can provide images captured by the processor (130) to manage the receipt and delivery of food.

[0086] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide angle of view, but is not limited thereto.

[0087] The memory (120) can store an operating system (OS) for controlling the overall operation of the components of the refrigerator (100), instructions related to the components of the refrigerator (100), a computer program including instructions, data on objects, or information on classification of objects.

[0088] In particular, the memory (120) can store various configurations for managing the receipt and delivery of food. In addition, the memory (120) can store a food database (DB) that stores information about food stored in the refrigerator (100) (e.g., type of food, capacity of food, expiration date of food, storage location of food, etc.).

[0089] Additionally, the memory (120) may, according to one embodiment, store a learned neural network model (e.g., a food classification model, etc.) for acquiring feature information corresponding to food items received or shipped from the refrigerator (100). Alternatively, the memory (120) may store a learned neural network model (e.g., a receipt / delivery recognition model, etc.) for recognizing food items received or shipped from the refrigerator (100).

[0090] However, it is not limited to this, and various neural network models, databases (DB), etc. may be stored in the memory (120) of the refrigerator (100) or may be stored in an external device such as a server.

[0091] The memory (120) may store a learned neural network model for converting photographed images of food items being stored in and out of the refrigerator (100). Alternatively, the memory (120) may store a learned neural network model (e.g., a feature extraction model or a feature matching model) for analyzing features of items being stored in and out of the refrigerator (100) and obtaining information on the classification of items being stored in and out of the refrigerator based on the analyzed features of the items.

[0092] Alternatively, in addition, the memory (120) may store images taken of the actual interior of the refrigerator (100) or objects being brought in and out. The memory (120) may be implemented in the form of memory embedded in the refrigerator (100) or may be implemented in the form of memory that can be detached and / or attached to the refrigerator (100) depending on the purpose of data storage.

[0093] For example, data for driving a refrigerator (100) may be stored in a memory embedded in the refrigerator (100), and data for expanding functions of the refrigerator (100) may be stored in a memory that can be detached and / or attached to the refrigerator (100).

[0094] In the case of memory embedded in the refrigerator (100), it may be implemented in the form of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0095] In the case of memory that can be detached and / or attached to the refrigerator (100), it can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to a USB (universal serial bus) port (e.g., USB memory), etc.

[0096] According to an embodiment, one or more processors (130) may control the overall operation of the refrigerator (100) according to at least one instruction stored in the memory (120). Specifically, one or more processors (130) may be connected to each component of the refrigerator (100) and control the overall operation of the refrigerator (100).

[0097] According to an embodiment, one or more processors (130) may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (TCON) for processing a digital signal. However, the present invention is not limited thereto, and one or more processors (130) may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an advanced RISC machine (ARM) processor, or an artificial intelligence (AI) processor, or may be defined by the terms thereof.

[0098] Additionally, one or more processors (130) may be implemented as a SoC (System on Chip), an LSI (Large Scale Integration) with a built-in processing algorithm, or may be implemented in the form of an FPGA (Field Programmable Gate Array). One or more processors (130) may perform various functions by executing computer executable instructions stored in the memory (120).

[0099] One or more processors (130) may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator.

[0100] One or more processors (130) may control one or any combination of other components of the electronic device and perform operations or data processing related to communication. One or more processors (130) may execute one or more programs or instructions stored in the memory (120). For example, one or more processors (130) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in the memory (120).

[0101] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors.

[0102] For example, when the first operation, the second operation, and the third operation are performed by the method according to the embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-only processor).

[0103] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores).

[0104] When one or more processors (130) are implemented as a multi-core processor, each of the plurality of cores included in the multi-core processor may include internal processor memory such as cache memory and on-chip memory, and a common cache shared by the plurality of cores may be included in the multi-core processor.

[0105] Each of the plurality of cores (or some of the plurality of cores) included in the multi-core processor may independently read and execute program instructions for implementing a method according to an embodiment of the present disclosure, or all (or some) of the plurality of cores may be linked to read and execute program instructions for implementing a method according to an embodiment of the present disclosure.

[0106] According to one embodiment of the present disclosure, one or more processors (130) may control the camera (110) to capture images of objects being stored or shipped into or out of the refrigerator (100). When one or more processors (130) acquire a captured image of an object being stored or shipped into the refrigerator (100), they may provide a guide for acquiring additional images based on the acquired captured image.

[0107] Here, "guide" may refer to instructions or guidance for capturing images. For example, the guide may include guidance information on the shooting angle, enabling users to obtain information about the classification of items being received or shipped based on the captured images. The guidance information included in the guide may be output as voice (or audio) or displayed as images or text.

[0108] When the side of a beverage can being put into or taken out of a refrigerator (100) is photographed, one or more processors (130) can guide the user to take a photograph of the front of the beverage can to obtain information about the classification of the beverage can (e.g., the brand, type, etc. of the beverage).

[0109] According to one embodiment of the present disclosure, one or more processors (130) may acquire additional images through the camera (110) when a signal for photographing an object corresponding to a provided guide is detected.

[0110] For example, when a user adjusts the position of an object so that the object is visible to the camera (110) according to a guide request (or a requested position, such as the front of the object), one or more processors (130) can detect the change in the position of the object as a signal to capture the object and obtain additional images of the object through the camera (110).

[0111] The camera (110) may include one or more cameras, and when multiple cameras are included, each camera (110) may capture images of objects being entered or exited from different locations. For example, when one or more processors (130) recognize the entry or exit of an object, they may capture images of the object from different directions through the multiple cameras (110) and register one or more captured images as data regarding the object being entered or exited.

[0112] According to one embodiment of the present disclosure, one or more processors (130) may store captured images and additional images as data regarding objects being received or shipped. For example, when one or more processors (130) acquire a captured image of a canned beverage, they may register an additional image for obtaining classification information about the canned beverage along with the captured image as data regarding the canned beverage.

[0113] Here, "data on items being stored in and out of the refrigerator" may refer to various types of information about items being stored in and out of the refrigerator. For example, data on items being stored in and out of the refrigerator (100) may include various types of data for obtaining information on the classification of items, such as captured images, additional images, generated images, and data on items acquired from external devices.

[0114] Here, "information about the classification of an object (or multiple objects)" may refer to various types of information about the classification of an object (or multiple objects). For example, information about the classification of an object may include at least one of the following: the name of the object, the type of object, the number of objects, information about purchasing ingredients, and information about the arrival or departure time of the object.

[0115] For example, one or more processors (130) may obtain information on shopping for purchasing food ingredients, etc., as information on classification of items by referring to data on items being received and shipped (e.g., information on a recent purchase list of food ingredients).

[0116] FIG. 3 is a detailed block diagram illustrating a refrigerator according to at least one embodiment of the present disclosure.

[0117] Referring to FIG. 3, a refrigerator (100) according to an embodiment of the present disclosure may include a camera (110), a memory (120), one or more processors (130), an output device (140), a communication device (150), a microphone (160), and a sensor (170). Parts that overlap with the above description will be omitted or abbreviated.

[0118] The output device (140) can provide various feedbacks. In particular, the output device (140) may include a speaker (141), a display (142), and light emitting diodes (LEDs) (143), as illustrated in FIG. 1 , or may further include other output devices (e.g., a haptic providing device, etc.), and any of the examples of the output device (140) described above may be omitted. However, this is merely an example and is not limited thereto.

[0119] At this time, a speaker (141) may be installed inside or outside the refrigerator (100) to provide various auditory feedback through audio. For example, if the captured image of food items being stored in the refrigerator is insufficient to obtain food information, one or more processors (130) may output audio such as "Please rotate the food slightly before taking the picture" through the speaker (141).

[0120] The LED (143) is installed in the storage compartment or inside the door of the refrigerator (100) and can provide various visual feedback through indicators of a specific shape and blinking, etc.

[0121] The display (142) may be positioned in at least a portion of one or more doors of the refrigerator to provide various visual feedback to the user. In particular, the display (142) may display a user registration UI (User Interface) requesting whether to save captured images in data regarding items being brought in and out.

[0122] The user registration UI may include an area where the user can directly register the brand name and type of the object in the photographed image of the object being stored in and out of the refrigerator (100). The user registration UI may be referred to by various terms such as user DB registration, user database lock, user registration information, etc.

[0123] The display (142) may display an object classification UI that includes information about object classification. For example, if two orange juices are stored in the refrigerator (100), the display (142) may display a UI that includes information about the product name of the orange juice, the date the orange juices were stored, the expiration date of the orange juice, and the number of orange juices remaining in the storage compartment.

[0124] The information contained in the object classification UI can be referred to by various terms, such as grocery information, grocery classification information, grocery list, and product information. These are described in detail in Figure 10.

[0125] The communication device (150) can communicate with an external server and / or an external terminal device. In particular, the communication device (150) can receive an image containing food or information about food from an external server to obtain information about food.

[0126] Additionally, the communication device (150) can receive images containing groceries or information about groceries, etc., to obtain information about groceries from the user terminal. The operation of performing communication with an external device is described in detail in FIG. 13.

[0127] The communication device (150) may include wired and / or wireless input and / or output interfaces (or input and / or output terminals) according to various standards. For example, one or more connection interfaces of the communication device (150) may include various interfaces such as HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), AP-based Wi-Fi (e.g., Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0128] The microphone (160) acquires audio signals and converts them into electrical signals, and may be installed inside or outside the refrigerator (100). In particular, the microphone (160) may receive audio signals including user voices. The user voices may include information regarding receipt or shipment and information regarding the classification of items (e.g., the type of food, the expiration date of the food, etc.).

[0129] The sensor (170) can detect the operating status of the refrigerator (100) (e.g., power or temperature) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. The sensor (170) can include one or more sensors, and a plurality of sensors can be placed at different locations within the refrigerator (100). In particular, one or more processors (130) can measure the temperature of one or more storage compartments of the refrigerator (100) using the sensing values ​​acquired through the sensors (170).

[0130] Alternatively, one or more processors (130) may recognize that a user is approaching through a sensor (170) and control the camera (110) to be ready. Alternatively, one or more processors (130) may detect that a door is open through a sensor (170) and operate the camera (110).

[0131] For example, one or more processors (130) may detect a signal for photographing an object corresponding to a guide provided based on the user's movement through a sensor (170). In this case, one or more processors (130) may control the camera (110) to obtain additional images of the object.

[0132] FIG. 4 is a diagram for explaining an operation of obtaining data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0133] One or more processors (130) can detect objects being entered or exited by a user's actions and obtain information on the classification of objects being entered or exited.

[0134] For example, when one or more processors (130) detect a user's movement of putting groceries (20) into a refrigerator (410), they can control a camera (110) to acquire a photographic image (21) of the groceries (20) being placed. The acquired photographic image can be input into a trained neural network model (e.g., a food classification model) (420).

[0135] If the result of the food classification model is recognized as not being a target of existing food classification or the result is not recognized as food (20) (430), the photographed image can be input into a learned neural network model (e.g., feature matching model) to compare the photographed image with previously stored data (e.g., data on items being received or shipped) to identify similarity (440).

[0136] One or more processors (130) can select data about objects being received or shipped that have a similarity level higher than a preset level with respect to a photographed image among the stored data about objects being received or shipped, and obtain information about object classification based on the data about the selected objects being received or shipped.

[0137] For example, if there is data on incoming or outgoing items whose similarity is higher than a preset level, one or more processors (130) can obtain product classification information such as “product name ABC and tomato juice” by using user DB (Database) Matching and “matching is successful” (451).

[0138] Conversely, if there is no data for incoming or outgoing items with a similarity level above a preset level, one or more processors (130) may "fail to match" and fail to obtain product classification information (452). However, this is merely an example, and the success or failure of the match may be determined by other criteria and / or methods.

[0139] If one or more processors (130) cannot obtain information on object classification based on the data on the selected object being received or shipped, they can provide a guide for obtaining additional images based on the obtained captured images.

[0140] In case one or more processors (130) fail to obtain information on object classification based on data on the selected object being received or shipped, they may directly request additional registration from the user in addition to providing guidance for obtaining additional images.

[0141] Specifically, the refrigerator (100) may request the user through the speaker (141) or the display (142) whether to store a photographed image in the data for the selected item being received or shipped, and when an input from the user corresponding to the request is received, the refrigerator (100) may proceed with additional registration by storing the photographed image in the data for the selected item being received or shipped.

[0142] Even if one or more processors (130) have obtained information about the classification of an object based on data about the selected object being received or shipped, they may perform an operation to obtain additional images if additional images are required.

[0143] For example, one or more processors (130) may provide guidance for acquiring additional images if the classification of incoming and outgoing food is incorrectly recognized or if the product matching results are incorrect. (For example, this may apply when information on the classification of objects obtained through a food classification model or feature extraction model does not match the incoming food (20).)

[0144] When a signal for photographing an object corresponding to a guide is detected, one or more processors (130) can acquire additional images via the camera (110) and store the captured images and additional images as data regarding the object being received or shipped. The operations for providing a guide for acquiring additional images and subsequent operations have been described above, so a redundant description thereof will be omitted.

[0145] One or more processors (130) can analyze the characteristics of objects being stored or shipped from the refrigerator (100) based on captured images. For example, captured images can be input into a feature analysis model to analyze the characteristics of objects being stored or shipped. However, this is not limited to this, and the feature analysis model can be included in various trained neural network models (e.g., feature extraction models).

[0146] One or more processors (130) can obtain information on the classification of objects being received or shipped based on the characteristics of the analyzed objects and store it in the memory (120).

[0147] FIG. 5 is a diagram for explaining an operation of acquiring data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0148] According to one embodiment, the refrigerator (100) can generate a plurality of sample images for generating guide images related to the acquired photographed images. Referring to FIG. 5, the refrigerator (100) analyzes the features of an image (31) of a canned beverage, and inputs data (31') obtained by analyzing the features of the photographed image (31) into a trained neural network model (500) to generate a plurality of sample images (32-1, 32-2, 32-3, 32-4).

[0149] According to one embodiment, a sample image that does not correspond to a captured image can be generated as a guide image based on the similarity between the generated plurality of sample images and the captured image. This will be described in detail in FIG. 6.

[0150] According to one embodiment, a plurality of sample images may be generated based on data regarding shooting angles for objects being received or shipped. The refrigerator (100) may obtain the angles at which objects enter or exit the actual refrigerator (100) based on images and angle values, and may generate images at shooting angles different from the obtained shooting angles as a plurality of sample images (32-1, 32-2, 32-3, 32-4).

[0151] For example, when the shooting angle of the image of the canned beverage is (15,0,50), a sample image (32-1) with a shooting angle different from the shooting angle of the captured image (e.g., (34,20,50) etc.) can be generated.

[0152] Here, the first number in the shooting angle represents the "Pan angle" by which the camera rotates left or right, and '15' in the shooting angle (15, 0, 50) could mean that the camera rotated 15 degrees to the right from the origin.

[0153] The second number in the shooting angle represents the "tilt angle", that is, the camera rotates up and down, and a '0' in a shooting angle of (15, 0, 50) could mean that the camera is not rotated in the vertical direction.

[0154] The third number in the shooting angle indicates the "zoom level," which is how far the camera zooms in or out. In a shooting angle of (15, 0, 50), "50" could mean that the captured image is zoomed in by about 50%.

[0155] FIG. 6 is a diagram for explaining an operation of obtaining data on an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0156] According to one embodiment, among a plurality of sample images (32-1, 32-2, 32-3, 32-4), a sample image (32-3) having a high similarity to data (31') obtained by analyzing the features of a photographed image (31) of a canned beverage can be removed from an additional photographing View Point. That is, based on the similarity between the photographed image and the plurality of sample images, a sample image that does not correspond to the photographed image (e.g., a sample image having a low similarity to data (31')) can be generated as a guide image.

[0157] For example, one or more processors (130) may analyze an image of an actual canned beverage and, if the shooting angle of the captured image is (15, 0, 50), remove a sample image (32-3) having a shooting angle of (15, 0, 50) from among a plurality of sample images from the guide image. One or more processors (130) may generate sample images (32-1, 32-2, 32-4) that do not have a shooting angle of (15, 0, 50) as guide images (33-1, 33-2, 33-4).

[0158] The refrigerator (100) can compare the numerical values ​​of the shooting angles of the plurality of sample images (32-1, 32-2, 32-3, 32-4) to identify the similarity between the data (31') obtained by analyzing the characteristics of the plurality of sample images (32-1, 32-2, 32-3, 32-4) and the shooting images.

[0159] For example, if the shooting angle of data (31') obtained by analyzing the features of the shooting image is (15, 0, 50), the sample image closest to "Pan angle 15, Tilt angle 0, Zoom level 50%" among the sample images can be identified as having a high similarity and removed from the guide image.

[0160] FIG. 7 is a drawing for explaining an operation of acquiring an image of an object being put into or taken out of a refrigerator according to at least one embodiment of the present disclosure.

[0161] According to one embodiment, the refrigerator (100) may distinguish categories of objects and request additional photos of required areas for each category of objects being brought in and out.

[0162] For example, if the refrigerator (100) detects that the object being brought in and out is a canned beverage (40), it may request additional image capture of the front (41-1), side (41-2), and back (41-3) as photos of the required area in the canned beverage category.

[0163] For example, if a user performs an action of putting a canned beverage into a refrigerator (100) and the front side (41-1) of the canned beverage is photographed, the refrigerator (100) may provide a guide in the form of a sound (e.g., voice) and / or image, such as “Please additionally photograph the side (41-2) and back (41-3).”

[0164] However, this is not limited to this, and additional images of various surfaces, such as the bottom, may be requested depending on the category of the object. For example, if a user places a chocolate with an expiration date written on the bottom into a refrigerator (100), the refrigerator (100) may provide guidance in the form of voice and / or images, such as, "Please also take an additional image of the bottom."

[0165] In another embodiment, if a user places an "orange" into the refrigerator (100), the refrigerator (100) can identify that the incoming food item is an orange based solely on a photographed image taken in a specific direction. If the refrigerator (100) can obtain information about the classification of incoming and outgoing items solely from the photographed image, it may not request additional photography from the user.

[0166] As described above, the refrigerator (100) may have images of areas requiring photography set in advance for each category of objects. However, this is not limited thereto, and the refrigerator (100) may analyze the characteristics of objects in real time while photographing objects being brought in and out, and provide a guide requesting images of the required areas.

[0167] FIG. 8 is a drawing for explaining a UI that provides a guide for a refrigerator according to at least one embodiment of the present disclosure.

[0168] According to one embodiment, the refrigerator (100) can generate guide images (53-1, 53-2, 53-3, 53-4) based on the acquired captured images. The user can register additional captured images corresponding to each guide image by clicking the capture button (801). When the registration progress of images requiring additional capture is 45% (806), registration information regarding the progress can be provided to the user through the UI.

[0169] If the refrigerator (100) identifies that the similarity between the guide image (53-1) and the corresponding photographed image (51-1) is above a preset level, it can provide feedback (802) to the user that the photograph was taken properly according to the guide.

[0170] In addition to the image actually captured, the refrigerator (100) can also use the guide image as a captured image (51-3) through user input, such as dragging. Similar to the feedback (802) indicating that the image was captured according to the guide, the refrigerator (100) can provide feedback (804) indicating that the guide image has been captured.

[0171] On the other hand, the refrigerator (100) can provide the user with feedback (805) requesting re-shooting along with feedback that the shot image (51-4) was not shot well according to the guide image (53-4) based on the identification of the similarity with which it corresponds.

[0172] When the refrigerator (100) corresponds to the guide image (53-2), if there is an image that has not yet been captured (51-2), it can provide feedback (803) to the user that the image has not yet been captured.

[0173] However, this is merely an example, and it is of course possible to register additional captured images and display the progress accordingly using other methods. Examples of display methods using other methods are described in detail in Fig. 9.

[0174] FIG. 9 is a drawing for explaining a UI for registering an additional image according to a guide of a refrigerator according to at least one embodiment of the present disclosure.

[0175] Referring to FIG. 9, if the refrigerator (100) is unable to obtain sufficient information on object classification based on the captured image, the refrigerator (100) may provide guidance to the user, such as "Turn the food slightly and take a picture" (901). For example, guidance may be provided, such as a specific angle at which the food should be rotated.

[0176] The user can click the shooting button (902) to take additional shots and register additional images (61-1, 61-2, 61-4). The refrigerator (100) can provide the user with guide images (63-1, 63-2, 63-2, 63-4) for additional shots until the additional registration is sufficient to obtain information about the classification of objects.

[0177] The refrigerator (100) can provide a UI to the user regarding the registration progress, taking into account the degree to which information on the classification of objects is sufficient and the degree of additional captured images.

[0178] For example, if there are four guide images for additional shooting, and three images are registered for shooting among the four guide images corresponding to each guide image, the refrigerator (100) can display information that the registration progress is 75% (903), for example, through a circular progress bar.

[0179] As described above, the refrigerator (100) can easily provide users with various information related to image capture and / or progress using a progress bar. However, this is not limited to this, and the method of providing information on registration progress can be varied.

[0180] The UI providing a guide for the refrigerator described in FIGS. 8 and 9 may be displayed on the display (142) of the refrigerator (100). However, the present invention is not limited thereto, and the refrigerator (100) may transmit information providing a guide for the refrigerator (100) to an external device to display the information on the display of the external device.

[0181] FIG. 10 is a diagram illustrating a UI that provides data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0182] According to one embodiment, when the refrigerator (100) acquires photographed images and additional images (71-1, 71-2, 71-3, 71-4) by the user, the refrigerator (100) can store the acquired images as data on objects being brought in and out and provide the user with information on object classification.

[0183] Referring to FIG. 10, information regarding the classification of an item may include information regarding the product name of the item, the name of the food item, and the entry date ([Brand Name], [Food / Ingredient Name], [Entered date]). However, this is not limited to this, and may include various other information, such as the time of shipment of the food item, the number of food items, or shopping information for the food item.

[0184] For example, the refrigerator (100) can obtain information (142') about the classification of the items, such as the product name "AA", the product type "Coffee", and the date of receipt "Today", based on the photographed images and additional images (71-1, 71-2, 71-3, 71-4), and display the information on the display (142).

[0185] In another embodiment, if the refrigerator (100) identifies a groceries with a product name of "AA" and a product type of "Coffee" as having been shipped, the information regarding the shipped groceries may be deleted from the UI displaying information regarding the product classification. However, if two or more identical groceries have been shipped, the information regarding the product classification may not be deleted, but rather the number of groceries remaining in the refrigerator (100) may be updated and displayed.

[0186] However, this is only one embodiment, and information on object classification may also be provided in the form of audio through a speaker (141). Here, the information on object classification may include at least one of the following: object name, object type, number of objects, and information on the time of receipt or shipment of objects, but is not limited thereto.

[0187] FIG. 11 is a diagram for explaining a UI that provides data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0188] According to one embodiment, the refrigerator (100) may display a list of captured images on the display (142) and provide them to the user in the form of a function that shows the list of captured images along with the inventory history or usage history. The user may directly register similar captured images as information about a specific food product by utilizing the function that shows the inventory history or usage history.

[0189] For example, if the images (142-1) taken at AM 01:40 and AM 01:39 are "AA Coffee", the user can select the images (142-1) taken at AM 01:40 and AM 01:39 and register the selected images as image data for "AA Coffee".

[0190] However, this is only one example, and the refrigerator (100) may automatically identify similar images (142-1) and display a UI requesting the user to register them in the food DB.

[0191] Although only a UI providing information on the classification of objects is illustrated in FIG. 10, the present invention is not limited thereto, and the refrigerator (100) may also display a UI including notification information on objects being brought in and out.

[0192] For example, the refrigerator (100) may analyze the receiving or shipping pattern of the received or shipped items based on information about the received or shipped items, and output notification information about the received or shipped items based on the analyzed receiving or shipping pattern through the speaker (141) and / or display (142).

[0193] Here, "notification information for items being received or shipped" may refer to various notification information related to the items being received or shipped. For example, notification information for items being received or shipped may include repurchase notifications for the item, notifications for similar items, consumption notifications for the item, and recipe notifications for the item. However, this is not limited to this, and may also include various information useful to the user based on the pattern of items being received or shipped.

[0194] The UI providing data on items being brought in and out as described in FIGS. 10 and 11 may be displayed on the display (142) of the refrigerator (100). However, the present invention is not limited thereto, and the refrigerator (100) may transmit data on items being brought in and out to an external device and display the data on items being brought in and out on the display of the external device.

[0195] FIG. 12 is a drawing for explaining an operation of extracting features for an object being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0196] According to one embodiment, the refrigerator (100) can input a photographed image into a learned neural network model (e.g., an image transformation model) and extract features of an object based on the output image.

[0197] For example, if the captured image is an image acquired from an external device and the image quality is higher than a preset level, the refrigerator (100) can input (1210) the captured image into an image conversion model and output (1220) the converted image as an image having a preset level of quality (e.g., a level of specifications supported by the refrigerator camera). The refrigerator (100) can extract (1230) the features of the object based on the converted image. An example of acquiring a captured image from an external device will be described in detail in FIG. 13.

[0198] Here, the "quality" of the image may include the image quality, brightness, features, etc. of the image. Accordingly, the refrigerator (100) may convert a high-quality image into a low-quality image, and then extract feature information about the object based on the converted low-quality image.

[0199] FIG. 13 is a diagram illustrating an example of a refrigerator receiving data from various electronic devices according to at least one embodiment of the present disclosure.

[0200] According to one embodiment, the refrigerator (100) can communicate with various external devices, such as a user terminal device and a server device, through a communication device (150). For example, the refrigerator (100) can receive an image captured by a user terminal device (200) and use the received image as a captured image.

[0201] The refrigerator (100) can communicate directly with a user terminal device (200) through a communication device (150), but this is only one embodiment, and it is of course possible to communicate with an external user terminal device through a server.

[0202] However, as described in FIG. 12, if the received image is identified as having a quality level higher than a preset level, the refrigerator (100) may use the received image in a form converted into an image with low quality.

[0203] In another embodiment, the refrigerator (100) may receive data about objects from the server device (300). For example, the refrigerator (100) may receive information about food images and product names of affiliated businesses from the server device (300). Accordingly, the refrigerator (100) may automatically register information about food items in the user food database based on the information received from the server device (300).

[0204] FIG. 14 is a flowchart illustrating an operation for managing data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0205] When an object is put into or taken out of the refrigerator (100), the refrigerator (100) can capture an image by taking a picture of the object (S1410). The refrigerator (100) can capture an image of the object by detecting the movement of the object being put into or taken out.

[0206] When the refrigerator (100) acquires a photographic image of an object being stored or shipped into the refrigerator, it can provide a guide for acquiring additional images based on the acquired photographic image (S1420). Since an embodiment of a method for providing a guide based on the acquired photographic image has been described above, a redundant description will be omitted.

[0207] When a signal for photographing an object corresponding to the provided guide is detected, the refrigerator (100) can acquire additional images (S1430). The refrigerator (100) can detect the movement of objects entering or leaving the refrigerator and perform additional photography. The refrigerator (100) can register the images acquired through the additional photography as additional images.

[0208] The refrigerator (100) can store captured images and additional images as data for items being received or shipped (S1440). The operation of the refrigerator (100) storing captured images and additional images as data for items being received or shipped may be included in the operation of registering additional images.

[0209] The refrigerator (100) can analyze the characteristics of objects based on data about objects being put in and out of the refrigerator (100), and obtain information about object classification based on the analyzed characteristics of the objects.

[0210] FIG. 15 is a flowchart illustrating an operation for managing data on objects being stored in or shipped from a refrigerator according to at least one embodiment of the present disclosure.

[0211] If the refrigerator (100) fails to obtain information on the classification of an object, it can identify similarity by comparing the captured image with data (e.g., previously stored data) regarding objects being received or shipped (S1510). The refrigerator (100) can identify similarity by comparing various characteristic information, such as the shape and angle of the object included in the image.

[0212] The refrigerator (100) can select data regarding incoming or outgoing objects whose similarity with the photographed image is above a preset level among data regarding incoming or outgoing objects (S1520). The refrigerator (100) can select data regarding incoming or outgoing objects with a high similarity level and identify the selected data as data regarding objects actually being incoming or outgoing.

[0213] The refrigerator (100) can obtain information on classification of items based on data on selected items to be received or shipped (S1530).

[0214] As described above, by obtaining information on the classification of items stored in and out of the refrigerator (100), the user can conveniently manage the food items stored in the refrigerator (100). Ultimately, this can improve the user experience.

[0215] The order of the flowcharts of the various embodiments described above is only one embodiment, and the order of each step of the flowchart may be changed, and of course, the order of the flowcharts may be performed simultaneously.

[0216] According to one embodiment of the present disclosure, one or more processors (130) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (120). The predefined operation rules or artificial intelligence models may be characterized as having been created through learning.

[0217] Here, "created through learning" means that a predefined set of behavioral rules and / or an artificial intelligence model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the artificial intelligence according to the present disclosure is implemented, or through a separate server / system.

[0218] An artificial intelligence model (e.g., a first object detection network and a second object detection network) may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of a previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer, and the neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0219] A learning algorithm is a method for training a target device using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0220] The control method described in FIGS. 14 and 15 can be performed by devices having various configurations, such as those of FIGS. 2 and 3 described above, but is not necessarily limited thereto, and can also be performed by devices having various configurations.

[0221] The various embodiments described above may be implemented as a single embodiment, or at least one of the embodiments may be combined in whole or in part to be implemented together in one device.

[0222] Various embodiments of the present disclosure may be implemented as software stored in a machine-readable storage media that can be installed or connected to a smartphone, a user terminal device, or other various electronic devices (e.g., a computer).

[0223] Specifically, a non-transitory readable storage medium may be provided in which software is stored for performing the steps of: when an object is received or shipped from a refrigerator, taking a picture of the object; when an image of the object being received or shipped from the refrigerator is obtained, providing a guide for obtaining an additional image based on the obtained image; when a signal for taking a picture of an object corresponding to the provided guide is detected, taking an additional image; and storing the taken image and the additional image as data for the object being received or shipped.

[0224] A device equipped with such a non-transitory readable medium can perform various operations, such as tag identification, confirmation of importance for each of multiple frames, and creation of an edited video, corresponding to the user actions described in the various embodiments described above.

[0225] In the context of non-transitory readable storage media, 'non-transitory' means that the storage medium does not contain only signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily on the storage medium.

[0226] Alternatively, a program for performing the method according to the various embodiments described above may be distributed online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0227] Each component (e.g., a module or a program) according to various embodiments may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., a module or a program) may be integrated into a single entity, which may perform the same or similar functions as the respective components prior to integration.

[0228] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0229] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the refrigerator, camera; At least one memory in which instructions are stored; and comprising at least one processor; At least one processor, By executing at least one of the above instructions individually or collectively, When an object is received or released from the refrigerator, the camera is controlled to take a picture of the object. When acquiring a photographic image of an object being stored or shipped from the refrigerator, a guide is provided for acquiring additional images based on the acquired photographic image. When a signal to photograph an object corresponding to the provided guide is detected, an additional image is acquired through the camera, A refrigerator that stores the above-mentioned photographed image and the above-mentioned additional image as data for the items being received or shipped.

2. In paragraph 1, At least one processor, By executing at least one of the above instructions individually or collectively, The above acquired photographed image is input into the learned neural network model, If it is identified that at least one piece of information for object classification cannot be obtained through the learned neural network model or that additional images are required, a guide is provided for obtaining the additional images. Information about the above classification of objects, Including at least one of the following information: the name of the thing, the type of the thing, the number of the things, and the time when the thing is received or shipped; If you identify that you need additional images, A refrigerator, including a case where the information on the object classification obtained through the learned neural network model is identified by the user as not corresponding to the object being received or shipped.

3. In paragraph 2, At least one processor, By executing at least one of the above instructions individually or collectively, If information on the classification of the above object is not obtained, the similarity between the stored data on the object and the obtained photographed image is identified, Identifying at least one data about the object among the previously stored data about the object, wherein the similarity is higher than a preset level, A refrigerator that obtains information about the classification of the object based on at least one of the identified data.

4. In paragraph 2, Speaker; and including display; At least one processor, By executing at least one of the above instructions individually or collectively, If information on the classification of the above object is not obtained, the similarity between the previously stored data on the object and the obtained photographed image is compared, Identifying at least one data about the object among the previously stored data about the object, wherein the similarity is higher than a preset level, Requesting the user through the at least one speaker or the display whether to store the captured image in the at least one data for the identified object, A refrigerator that, when a user's input is received, stores the captured image in at least one of the identified data for the object.

5. In paragraph 2, Speaker; and including display; At least one processor, By executing at least one of the above instructions individually or collectively, Requesting the user via at least one speaker or the display whether to save the above-described captured image to previously stored data, A refrigerator that, when a user's input is received, stores the photographed image in at least one of the data.

6. In paragraph 1, Speaker; and including display; At least one processor, By executing at least one of the above instructions individually or collectively, The acquired above-mentioned photographed image is input into the learned neural network model, When information on object classification is obtained through the learned neural network model, the information on object classification is stored in at least one memory, Outputting information about the above object classification through at least one speaker or display, Information about the above classification of objects, A refrigerator comprising at least one of information on the name of the object, the type of the object, the number of the object, and the time of receipt or shipment of the object.

7. In paragraph 1, The above guide includes at least one guide image, At least one processor, By executing at least one of the above instructions individually or collectively, Generating a plurality of sample images for generating at least one guide image based on the acquired photographed image, Based on the similarity between the acquired photographed image and each image among the plurality of sample images, a sample image that does not correspond to the acquired photographed image is generated as at least one guide image, Each image among the above multiple sample images is, A refrigerator generated based on data on the shooting angle of the above-mentioned items being received or shipped.

8. In paragraph 1, At least one processor, By executing at least one of the above instructions individually or collectively, Based on the acquired photographed images, the characteristics of objects being stored in or shipped from the refrigerator are analyzed, Obtain information on the classification of the items being received or shipped based on the characteristics of the items analyzed, A refrigerator that stores information on the classification of the items to be received or shipped in the at least one memory.

9. In paragraph 1, further comprising a communication device configured to communicate with an external device; The above acquired photographed images are, Based on at least one of the images acquired through the camera or the images acquired from the external device through the communication device, At least one processor, By executing at least one of the above instructions individually or collectively, A refrigerator that converts an image obtained from the external device into an image having a quality level higher than a preset level.

10. In paragraph 1, Speaker; and including display; At least one processor, By executing at least one of the above instructions individually or collectively, Based on the information about the items being received or shipped, the receiving or shipping pattern for the items being received or shipped is analyzed, Based on the analyzed pattern of receipt or shipment, notification information about the items being received or shipped is output through at least one speaker or the display, Notification information for the above items is: A refrigerator comprising at least one of repurchase information for the object, information on objects similar to the object, consumption information for the object, or recipe information for the object.

11. In the method of controlling a refrigerator, When an object is received or delivered from the refrigerator, a step of taking a picture of the object; A step of obtaining a photographic image of an object being stored in or shipped from the refrigerator, and providing a guide for obtaining an additional image based on the obtained photographic image; When a signal for photographing the object corresponding to the provided guide is detected, a step of acquiring an additional image; and A control method, comprising: a step of storing the above-described photographed image and the above-described additional image as data for the object being received or shipped; 12. In paragraph 11, The step of providing a guide for photographing the above objects is, A step of inputting the acquired photographed image into a learned neural network model; and A step of providing a guide for capturing the additional image when it is identified that at least one piece of information for object classification cannot be obtained through the learned neural network model or that an additional image is required; Information about the above classification of objects, Including at least one of the following information: the name of the thing, the type of the thing, the number of the things, and the time when the thing is received or shipped; If you identify that you need additional images, A control method, including a case where the information on the object classification obtained through the learned neural network model is identified by the user as not corresponding to the object being received or shipped.

13. In paragraph 12, If information on the classification of the object cannot be obtained, a step of identifying similarity by comparing the previously stored data on the object with the obtained photographed image; A step of identifying at least one data about the object among the previously stored data about the object, the similarity of which is higher than a preset level; and A control method further comprising: a step of obtaining information about the classification of the object based on the at least one piece of data about the identified object.

14. In paragraph 12, If information on the classification of the object is not obtained, a step of comparing the similarity between the previously stored data on the object and the captured image; A step of identifying at least one data about the object among the previously stored data about the object, the similarity of which is higher than a preset level; A step of requesting a user whether to store the photographed image in the at least one data for the identified incoming or outgoing object; and A control method further comprising: when a user's input is received, storing the captured image in at least one of the identified data for the object.

15. In paragraph 12, A step of requesting a user through at least one speaker or the display whether to save the captured image to previously stored data; and A control method further comprising: when a user's input is received, storing the photographed image in at least one data;

Images