Shopping information providing method and refrigerator using same

Abstract

Provided are a shopping information providing method and a refrigerator using the method. Specifically, the method includes the operations of: identifying a category of displayed food items on a display stand in front of which a user is located in a store; generating shopping information related to stored food items included in the identified category among one or more food items stored in an in-house refrigerator; and transmitting the generated shopping information to a user terminal. The shopping information includes at least one of a list of the stored food items, locations of the stored food items, an internal image displaying the stored food items, a required list requiring purchase in the category, recipe information related to the displayed food items and the stored food items, and recommended storage location information related to where the displayed food items are to be stored in the refrigerator.

Description

Method for providing shopping information and refrigerator applying the method

[0001] The present disclosure relates to a method for providing shopping information and a refrigerator employing the method. Specifically, the present disclosure relates to a technology for providing shopping information based on food information in a refrigerator to a user terminal when a user enters a store to shop for food.

[0002] Advances in Internet of Things (IoT) technology have enabled home appliances like refrigerators to be wirelessly connected to user devices. Refrigerators can then provide information about the food stored in them to user devices.

[0003] Additionally, a user terminal can register a zone that includes a specific location, such as a home. When the user terminal's current location enters the zone, the user terminal can request information from home appliances. Home appliances can then provide the user terminal with information related to themselves. For example, when the user terminal enters the zone, a refrigerator can provide the user terminal with information related to food stored in the refrigerator.

[0004] However, the existing method requires users to register their area in advance and may not be available if the user is outside the registered area. For example, if a user is shopping for groceries outside the registered area, they cannot receive food-related information from the refrigerator.

[0005] Additionally, the existing method may reduce the user's shopping efficiency because it provides information only by considering whether the user has entered the area, regardless of whether the user currently intends to shop for food or whether the user is in a situation to purchase the food they need.

[0006] A method for providing shopping information according to one embodiment of the present disclosure includes an operation of identifying a category of food sold in a store, an operation of generating shopping information related to stored food included in the identified category among at least one food stored in a refrigerator at home, and an operation of transmitting the generated shopping information to a user terminal, wherein the shopping information may include at least one of a list of stored food, a location of stored food, an internal image of a refrigerator displaying stored food, a list of items that need to be purchased in the category, recipe information related to sold food and stored food, and recommended storage location information related to where to store sold food in the refrigerator.

[0007] A refrigerator applying a shopping information providing method according to one embodiment of the present disclosure includes a communication module, at least one camera, and at least one processor, wherein the at least one processor receives a result of identifying a category of food sold in a store from a user terminal, generates shopping information related to stored food included in the identified category among at least one stored food, and transmits the generated shopping information to the user terminal, and the shopping information may include at least one of a list of stored food, a location of stored food, an internal image of the refrigerator displaying the stored food, a list of items that need to be purchased in the category, recipe information related to the sold food and stored food, and recommended storage location information related to where to store the sold food in the refrigerator.

[0008] FIG. 1 is a diagram illustrating a system for implementing a method for providing shopping information according to one embodiment of the present disclosure.

[0009] FIG. 2 is a drawing showing the external structure of a refrigerator according to one embodiment of the present disclosure.

[0010] FIG. 3 is a drawing showing the internal structure of a refrigerator according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating a method for providing shopping information according to one embodiment of the present disclosure.

[0012] FIG. 5 is a drawing showing a refrigerator according to one embodiment of the present disclosure displaying information on food stored in the refrigerator.

[0013] FIG. 6 is a drawing showing a refrigerator according to one embodiment of the present disclosure displaying information on food stored in the refrigerator.

[0014] FIG. 7 is a drawing showing a refrigerator according to one embodiment of the present disclosure displaying a record of food being brought into and taken out of the refrigerator.

[0015] FIG. 8 is a flowchart specifically illustrating a method for identifying categories of food displayed in a store to provide shopping information according to one embodiment of the present disclosure.

[0016] FIG. 9 is a diagram illustrating obtaining store information based on map information to identify a category according to one embodiment of the present disclosure.

[0017] FIG. 10 is a diagram illustrating obtaining store information based on map information to identify a category according to one embodiment of the present disclosure.

[0018] FIG. 11 is a diagram illustrating obtaining shelf information based on indoor information to identify a category according to one embodiment of the present disclosure.

[0019] FIG. 12 is a diagram illustrating identification of a category based on a display image captured by a user terminal according to one embodiment of the present disclosure.

[0020] FIG. 13 is a flowchart specifically illustrating a method for obtaining information related to food included in an identified category to provide shopping information according to one embodiment of the present disclosure.

[0021] FIG. 14a is a diagram illustrating outputting an internal image corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0022] FIG. 14b is a diagram illustrating outputting an internal image corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0023] FIG. 15 is a diagram showing an internal image corresponding to a category according to one embodiment of the present disclosure.

[0024] FIG. 16 is a diagram illustrating outputting storage list information and required list information corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0025] FIG. 17 is a diagram illustrating outputting recipe information corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0026] FIG. 18 is a diagram illustrating outputting recommended storage locations of displayed food to provide shopping information according to one embodiment of the present disclosure.

[0027] FIG. 19 is a diagram illustrating a UI when a food category is not identified when entering a store to provide shopping information according to one embodiment of the present disclosure.

[0028] FIG. 20 is a diagram illustrating a UI when a category of food is identified upon entering a store to provide shopping information according to one embodiment of the present disclosure.

[0029] FIG. 21 is a diagram illustrating a UI when a category of food is identified based on user input to provide shopping information according to one embodiment of the present disclosure.

[0030] FIG. 22 is a diagram illustrating a UI when individual foods are identified based on user input to provide shopping information according to one embodiment of the present disclosure.

[0031] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or alternatives of the embodiments.

[0032] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0033] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0034] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0035] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0036] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0037] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0038] Terms such as "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0039] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0040] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0041] A refrigerator according to one embodiment may include a body.

[0042] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0043] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0044] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0045] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0046] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0047] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0048] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0049] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0050] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0051] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0052] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0053] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door basket for storing items.

[0054] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0055] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0056] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0057] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0058] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment through heat generation and cooling through the Peltier effect.

[0059] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0060] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0061] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0062] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0063] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0064] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0065] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0066] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0067] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0068] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0069] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0070] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) where the refrigerator or user device is connected to the wide area network (WAN) where the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0071] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0072] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0073] A method for providing shopping information according to the present disclosure and a system implementing the method are intended to display shopping information through a mobile device based on food information in a refrigerator when a user enters a store for grocery shopping.

[0074] Hereinafter, with reference to the attached drawings, a shopping information providing method according to the present disclosure and a system implementing the method are specifically described.

[0075] FIG. 1 is a diagram illustrating a system for implementing a method for providing shopping information according to one embodiment of the present disclosure. The system according to one embodiment may include a user terminal (112, 113) carried by a user (111) located in a store (110), a refrigerator (121, 122) located in a home (120), and a server (130).

[0076] A store (110) may be a place where a user (111) wishes to purchase food. For example, the store (110) may be at least one of a grocery store, a supermarket, a butcher shop, and a mart.

[0077] The user terminal (112, 113) may be an electronic device carried by the user (110). For example, the user terminal (112, 113) may be a smart phone (112). However, the present invention is not limited thereto, and the user terminal (112, 113) may also be a smart watch (113). For convenience of explanation, the reference number of the user terminal (112, 113) may be referred to as the user terminal (112) below.

[0078] The user terminal (112) can communicate with the server (130) or the refrigerator (121, 122) to receive information on food stored in the refrigerator (121, 122). The user terminal (112) can recognize that the user has entered the store (110) using location information (GPS).

[0079] If a location-recognition beacon is installed within a store (110), the user terminal (112) can recognize the shelf or corner where the user is located based on a signal received from the beacon. For example, the beacon may be a wireless communication AP, an optical signal transceiver, or an acoustic signal transceiver. The user terminal (112) can identify the category of food displayed on the shelf or corner based on the recognized shelf or corner.

[0080] The user terminal (112) can identify the category of food by recognizing an image within the store (110). The user terminal (112) can identify the category of food by recognizing the product code of the food. For example, the product code of the food may be a QR code or barcode printed on the food packaging. The user terminal (112) can identify the category of food by receiving user input.

[0081] A refrigerator (121, 122) may be a home appliance for storing food. The refrigerator (121, 122) may be a smart refrigerator (121) or a smart pantry (122). For convenience of explanation, the reference number of the refrigerator (121, 122) may be referred to as refrigerator (121) below.

[0082] The refrigerator (121) can recognize food being stored or in / out of food. The refrigerator (121) can generate food information including a list of stored food and records of in / out food. The refrigerator (121) can be wirelessly connected to a server (130) and a user terminal (112). The refrigerator (121) can transmit food information to the server (130). The refrigerator (121) can transmit food information to the user terminal (112) via the server (130).

[0083] The server (130) may be an Internet of Things (IoT) server. The server (130) may wirelessly connect the refrigerator (121) and the user terminal (112). The server (130) may transmit information related to the category of food identified by the user terminal (112) to the refrigerator (121). The server (130) may transmit food information transmitted from the refrigerator (121) to the user terminal (112). The server (130) may store food information received from the refrigerator (121). For example, the server (130) may receive and store an internal image of the refrigerator (121) from the refrigerator (121).

[0084] The server (130) of the present disclosure may be an entity that provides shopping information to a user terminal (112) based on food information received from a refrigerator (121). The server (130) may generate shopping information requested by the user terminal (112) using the stored food information. For example, the server (130) may generate shopping information related to a category of food identified by the user terminal (112) using the stored internal image. The server (130) may transmit the generated shopping information to the user terminal (112).

[0085] The server (130) may request the refrigerator (121) for information necessary to generate shopping information requested by the user terminal (112). For example, if the server (130) requires a current internal image of the refrigerator (121) to generate shopping information, the server (130) may request the refrigerator (121) to provide the internal image of the refrigerator (121) in real time. The server (130) may receive the necessary information from the refrigerator (121) and generate the shopping information requested by the user terminal (112).

[0086] FIG. 2 is a drawing showing the external structure of a refrigerator (121) according to one embodiment of the present disclosure. The refrigerator (121) according to one embodiment may include a main body (210), at least one door (220, 230, 240, 250), a hinge (260), a display (270), an external camera (280), and a communication module (290).

[0087] The main body (210) can form the exterior of the refrigerator (121). The main body (210) can prevent cold air inside the refrigerator (121) from leaking out to the outside. The main body (210) can prevent warm air outside the refrigerator (121) from flowing into the inside. The main body (210) can include an insulating material that prevents the inflow and outflow of cold and warm air of the refrigerator (121).

[0088] At least one door (220, 230, 240, 250) can open and close the front side of the main body (210). At least one door (220, 230, 240, 250) can open or close the main body (210) by rotating around the front edge of the main body (210) via a hinge (260). For example, at least one door (220, 230, 240, 250) can be arranged at the front upper left side of the main body (210), the front upper right side of the main body (210), the front lower left side of the main body (210), and the front lower right side of the main body (210). However, it is not limited thereto, and the number of at least one door (220, 230, 240, 250) and the arrangement position of at least one door (220, 230, 240, 250) may vary depending on the design of the refrigerator (121).

[0089] A hinge (260) can connect a body (210) and at least one door (220, 230, 240, 250). The hinge (260) can rotate at least one door (220, 230, 240, 250) by a set angle.

[0090] The display (270) can visually display information related to food stored in the refrigerator (121). For example, the display (270) can display a list of food stored in the refrigerator (121). For example, the display (270) can display a record of food entering and leaving the refrigerator (121).

[0091] The external camera (280) can capture the front exterior of the refrigerator (121). The external camera (280) can monitor the front exterior of the refrigerator (121).

[0092] The communication module (290) can be connected to a network. The communication module (290) can communicate with a user terminal (112) or a server (130).

[0093] FIG. 3 is a drawing showing the internal structure of a refrigerator (121) according to one embodiment of the present disclosure. The refrigerator (121) according to one embodiment may include a storage compartment (310) and at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343).

[0094] A storage compartment (310) may be formed inside the main body (210). The storage compartment (310) may store food in a refrigerated or frozen state. The storage compartment (310) may include a refrigerator compartment (311) and a freezer compartment (312). The refrigerator compartment (311) may be a location where food is kept cold at a temperature above freezing. The freezer compartment (312) may be a location where food is kept frozen at a temperature below freezing.

[0095] At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) may be positioned on at least one of the inner upper surface of the main body (210), the inner side surface of the main body (210), and at least one door (220, 230, 240, 250). For example, among the at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343), the inner upper camera (321, 322, 323) may be positioned on the inner upper surface of the main body (210). For example, at least one internal left camera (331, 332, 333) among at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) may be positioned on the internal left side of the main body (210) or on at least one left door (220, 240). For example, at least one internal right camera (341, 342, 343) among at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) may be positioned on the internal right side of the main body (210) or on at least one right door (230, 250).

[0096] At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can photograph the inside of the refrigerator (121). At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can photograph food being put into or taken out of the refrigerator (121). At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can photograph a user putting food into or taking out of the refrigerator (121). For example, at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can photograph the storage room (310). For example, at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can photograph a door bin, which is a shelf provided inside at least one door (220, 230, 240, 250).

[0097] At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can form a photographing area to look at at least one door (220, 230, 240, 250) from the top of the refrigerator (121). At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can monitor the photographing area to identify whether a user is putting in or taking out food. At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) can take a photograph of the photographing area when it recognizes that a user is putting in or taking out food.

[0098] FIG. 4 is a flowchart illustrating a method for providing shopping information according to one embodiment of the present disclosure.

[0099] In operation 410, a system for providing shopping information according to an embodiment of the present disclosure can identify the category of food sold in a store. The system for providing shopping information according to an embodiment may include a user terminal. For example, the user terminal can identify the type of store in which the user is located and the category of food sold in the store. For example, the user terminal can identify the shelf in which the user is located in the store and the food sold on the shelf in which the user is located. Based on the identification result, the user terminal can identify the category of the food sold. For example, based on the identification result, the user terminal can identify the category of the food sold as vegetables.

[0100] In operation 420, a system for providing shopping information according to an embodiment of the present disclosure may generate shopping information related to stored food belonging to an identified category among at least one food stored in a home refrigerator. The refrigerator or server may determine at least one stored food belonging to an identified category of sale food as stored food. For example, the refrigerator may determine a vegetable belonging to a category of sale food among at least one stored food as stored food. In this case, the refrigerator may generate shopping information related to the vegetable. For example, the server may determine a vegetable belonging to a category of sale food among at least one stored food as stored food. The server may receive a list of foods stored in the refrigerator from the refrigerator. From the received list of foods, the server may determine a vegetable belonging to a category of sale food as stored food.

[0101] Shopping information can include a list of stored foods. The refrigerator can generate a list of stored foods that fall into the category identified as the food for sale. For example, if the category of the food for sale is vegetables, the refrigerator can generate a list of foods stored in the refrigerator that fall into the vegetable category, such as onions, carrots, and tomatoes.

[0102] Shopping information can include the location of stored food. The refrigerator can generate information related to the storage locations of food belonging to the category identified as the food for sale. For example, if the category of the food for sale is vegetables, the refrigerator can generate information indicating where onions, carrots, and tomatoes, which are classified as vegetables, are stored within the refrigerator. The refrigerator can also generate information related to the possible storage locations of food belonging to the category identified as the food for sale. For example, if the category of the food for sale is vegetables, the refrigerator can generate information indicating where onions, carrots, and tomatoes, which are classified as vegetables, are stored within the refrigerator.

[0103] Shopping information may include an internal image indicating the stored food. The refrigerator may obtain an internal image of the food stored in the category identified as the food for sale. For example, if the category of the food for sale is vegetables, the refrigerator may obtain an internal image of the area where onions, carrots, and tomatoes, which are vegetables, are stored. The refrigerator may generate information related to the locations in the refrigerator where the food for sale can be stored. For example, if the category of the food for sale is vegetables, the refrigerator may generate information indicating where onions, carrots, and tomatoes, which are vegetables, can be stored among the food for sale.

[0104] Shopping information can include a needs list for purchases within a category. The refrigerator can determine the inventory of stored food and generate a needs list listing items that need to be purchased within the category identified as the food being sold. For example, if the category of the food being sold is vegetables, the refrigerator can determine the inventory of vegetables stored in the refrigerator and generate a needs list listing items that need to be purchased, such as cucumbers and potatoes, which are out of stock.

[0105] Shopping information can include recipe information related to the food on sale and stored in the refrigerator. The refrigerator can generate recipes for dishes that can be made using the food on sale and stored in the refrigerator. For example, if the food on sale is beef, the refrigerator can generate a recipe for making bulgogi using the beef and onions stored in the refrigerator.

[0106] Shopping information can include recommended storage locations for food items within the refrigerator. The refrigerator can generate information including recommended storage locations for food items identified in the store. For example, if the food item being sold is tomatoes, the refrigerator can generate information recommending the fresh food compartment as the appropriate storage location for tomatoes.

[0107] In operation 430, a system for providing shopping information according to an embodiment of the present disclosure may transmit the generated shopping information to a user terminal. The refrigerator or server may provide the user terminal with shopping information related to food belonging to an identified category. For example, the refrigerator may generate shopping information related to food it stores belonging to an identified category and provide the shopping information to the user terminal. For example, the server may request information related to food stored in the refrigerator belonging to an identified category from the refrigerator. The server may generate shopping information based on the information related to the food stored in the refrigerator received from the refrigerator and provide the shopping information to the user terminal.

[0108] A user terminal can receive shopping information from a refrigerator or a server. The user terminal can display the shopping information received from the refrigerator or the server. The shopping information can include information related to food stored in the refrigerator corresponding to the category to which the displayed food belongs. Accordingly, a system for providing shopping information according to an embodiment of the present disclosure can automatically provide information related to food currently stored in the refrigerator or food to be purchased to a user who is shopping for food without any separate operation, thereby efficiently preventing duplicate purchases or omissions of food. In addition, a system for providing shopping information according to an embodiment of the present disclosure can determine the category of displayed food displayed in a store or on a shelf where the user is currently located and provide shopping information corresponding to the identified category, thereby providing shopping information tailored to the user's location and movement within the store, thereby improving the user's shopping efficiency within the store.

[0109] FIG. 5 is a diagram illustrating a refrigerator (121) according to an embodiment of the present disclosure displaying food information (510) stored in the refrigerator (121). FIG. 6 is a diagram illustrating a refrigerator (121) according to an embodiment of the present disclosure displaying food information (610) stored in the refrigerator (121). FIGS. 5 and 6 may be specific examples included in an embodiment illustrating a refrigerator (121) displaying food information. For example, when displaying food information, the refrigerator (121) may display the information in a form that includes all of the information illustrated in FIGS. 5 and 6, may display the information in a form similar to FIG. 5, or may display the information in a form similar to FIG. 6. However, the present invention is not limited thereto, and the refrigerator (121) may display information related to food stored in the refrigerator (121) in a form that can be typically derived in addition to the forms illustrated in FIGS. 5 and 6.

[0110] The refrigerator (121) or server (130) can recognize food stored in the refrigerator (121). The refrigerator (121) or server (130) can generate food information (510) including a list of stored foods. The refrigerator (121) can display the generated food information (510) through a display (170). The food information (510) can be stored in the refrigerator (121) or server (130).

[0111] Food information (510) may include the name of the food stored in the refrigerator (121), an image of the food stored in the refrigerator (121), and the last time the food stored in the refrigerator (121) was received. The image of the food stored in the refrigerator (121) may be an image acquired when the food is received in the refrigerator (121). For example, the image of the food stored in the refrigerator (121) may be an image captured by at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343) of the refrigerator (121) when the food is received in the refrigerator (121).

[0112] Food information (510) may include a plurality of icons (511, 512, 513, 514, 515, 516). For example, a first icon (511) included in food information (510) may display text indicating that the stored food is banana, an image of a banana, and information indicating that the banana was last received on January 4th. For example, a second icon (512) included in food information (510) may display text indicating that the stored food is spinach, an image of spinach, and information indicating that the spinach was last received on January 5th. For example, a third icon (513) included in food information (510) may display text indicating that the stored food is juice, an image of juice, and information indicating that the juice was last received on January 6th. For example, the fourth icon (514) included in the food information (510) may display text indicating that the stored food is a tomato, an image of a tomato, and information that the tomato was last received on January 7. For example, the fifth icon (515) included in the food information (510) may display text indicating that the stored food is an apple, an image of an apple, and information that the apple was last received on January 8. For example, the sixth icon (516) included in the food information (510) may display text indicating that the stored food is cheese, an image of cheese, and information that the cheese was last received on January 9.

[0113] The refrigerator (121) or server (130) can recognize food stored in the refrigerator (121). The refrigerator (121) or server (130) can generate food information (610) including a list of stored foods. The refrigerator (121) can display the generated food information (610) through a display (170). The food information (610) can be stored in the refrigerator (121) or server (130).

[0114] Food information (610) may display the name of the food stored in the refrigerator (121), the expiration date of the food stored in the refrigerator (121), and the storage location of the food stored in the refrigerator (121). For example, a first list (611) included in the food information (610) may display text indicating that the stored food is soup, text indicating that the soup has an expiration date of January 10, and information indicating that the soup is stored in the main storage room on the left. For example, a second list (612) included in the food information (610) may display text indicating that the stored food is beef, text indicating that the beef has an expiration date of January 12, and information indicating that the beef is stored in the main storage room on the right. For example, a third list (613) included in the food information (610) may display text indicating that the stored food is onion, text indicating that the onion has an expiration date of January 13, and information indicating that the onion is stored in the fresh storage room on the left. For example, a fourth list (614) included in food information (610) may display text indicating that the stored food is milk, text indicating that the milk has an expiration date of January 15th, and text indicating that the milk is stored in the right door. For example, a fifth list (615) included in food information (610) may display text indicating that the stored food is grapes, text indicating that the grapes have an expiration date of January 17th, and text indicating that the grapes are stored in the right fresh storage room. For example, a sixth list (616) included in food information (610) may display text indicating that the stored food is cheese, text indicating that the cheese has an expiration date of January 20th, and text indicating that the cheese is stored in the left door.

[0115] FIG. 7 is a drawing showing a refrigerator (121) according to one embodiment of the present disclosure displaying a record (710) of food being brought into and taken out of the refrigerator (121).

[0116] The refrigerator (121) or server (130) can recognize food being stored in or out of the refrigerator (121). The refrigerator (121) or server (130) can generate an entry / exit record (710) based on the recognition result. The refrigerator (121) can display the generated entry / exit record (710) through the display (170). The entry / exit record (710) can be stored in the refrigerator (121) or server (130).

[0117] The receipt / distribution record (710) may display the name of the food that was received into or taken out of the refrigerator (121), a classification of whether the food was received into or taken out of the refrigerator (121), and the time the food was received into or taken out of the refrigerator (121). For example, the first record (711) included in the receipt / distribution record (710) may display that a sandwich was taken out of the refrigerator (121) at 7:00 PM on January 20, and that beef was received into the refrigerator (121) at 4:00 PM on January 20. For example, the second record (712) included in the receipt / disbursement record (710) may indicate that onions were taken out of the refrigerator (121) at 3:00 AM on January 19, and that milk was received into the refrigerator (121) at 11:00 AM on January 19. For example, the third record (713) included in the receipt / disbursement record (710) may indicate that bananas were received into the refrigerator (121) at 5:00 PM on January 18, and that eggs were taken out of the refrigerator (121) at 9:00 AM on January 18. The receipt / disbursement record (710) may also indicate the user who received or removed food from the refrigerator (121).

[0118] In one embodiment, a refrigerator (121) or a server (130) may generate a list of foods to be purchased. The refrigerator (121) or the server (130) may generate the list of foods to be purchased based on a list of foods stored in the refrigerator (121), a record of receipts and withdrawals (710), and planned meal information. For example, the refrigerator (121) or the server (130) may add a food that has been received and withdrawn more than a certain number of times and is not in the list of currently stored foods to the list of foods to be purchased. For example, the refrigerator (121) or the server (130) may add a food that is required for a diet but is not in the list of currently stored foods to the list of foods to be purchased. For example, the refrigerator (121) or the server (130) may add a food that is expected to be consumed soon to the list of foods to be purchased by considering the record of receipts and withdrawals (710) among the currently stored foods. For example, a refrigerator (121) or server (130) may add food items whose expiration date is imminent among the currently stored foods to a list of foods that need to be purchased. The list of foods that need to be purchased may be stored in the refrigerator (121) or server (130).

[0119] FIG. 8 is a flowchart specifically illustrating a method for identifying categories of food displayed in a store to provide shopping information according to one embodiment of the present disclosure.

[0120] In operation 810, a system for providing shopping information according to an embodiment of the present disclosure can identify that a user has entered a store based on at least one of location information and map information of a user terminal. The user terminal can obtain the user's location information and map information using location measurement technology such as GPS or geo-fencing technology. The user terminal can recognize that the user has entered the store using the location information and map information. The store may be a food-only store or a general store. For example, the store may be a food-only store such as a butcher shop, fish shop, vegetable shop, or fruit shop. For example, the store may be a general store such as a department store, large discount store, or supermarket.

[0121] In operation 820, a system for providing shopping information according to an embodiment of the present disclosure may acquire at least one of information from the user's location information, information related to displayed food, and information related to a display shelf where the displayed food is placed. Displayed food may refer to food displayed on a display shelf within a store among foods sold within the store. Since foods sold in a store are sold to users after being placed on a display shelf, the term "displayed food" may be used interchangeably with "sold food" hereinafter. The user terminal may acquire at least one of information from the type of store, information on the corner of the store, information on the display shelf of the store, and the detailed location of the user. The user terminal may identify a category of food based on at least one of the type of store, information on the corner of the store, information on the display shelf of the store, and the detailed location of the user. For example, the user terminal may recognize the type of store where the user is located. For example, the user terminal may recognize the display shelf or corner where the user is located within the store. However, the present invention is not limited thereto, and the user terminal may obtain at least one of the user's location information, information related to the displayed food, and information related to the display shelf on which the displayed food is placed, using various methods such as image capturing, map information, indoor positioning, code recognition, text input, or voice input.

[0122] In operation 830, a system for providing shopping information according to an embodiment of the present disclosure can identify a category based on at least one piece of information acquired. The category of food recognized in a store can be identified based on at least one of the following: user location information, information related to displayed food, and information related to the display shelf on which the displayed food is placed. For example, the user terminal can determine the category of food based on the result of recognizing the type of store the user is located in. For example, the user terminal can determine the category of food based on the result of recognizing the display shelf or corner where the user is located within the store.

[0123] FIG. 9 is a diagram illustrating obtaining store information based on map information to identify a category according to one embodiment of the present disclosure.

[0124] The user terminal can obtain the user's location information and map information. For example, the user terminal can obtain a street map (910) and the user's location on the street (920). The street map (910) can include information related to the user's route, the location of stores, the entrances to each store, the size of each store, and the type of each store.

[0125] The user terminal can recognize that the user has entered a store based on location information and map information. For example, the user terminal can recognize that the user has entered the entrance of a store (930) located in the center left of the street map (910) based on the street map (910) and the user's location (920) on the street.

[0126] The user terminal can use location and map information to identify whether the store the user has entered is a dedicated store handling a specific category of food. For example, based on a street map (910) and the user's location on the street (920), the user terminal can identify that the store (930) the user has entered is a butcher shop handling meat.

[0127] If the store the user enters is a dedicated store, the user terminal can identify the food category based on the store type. For example, if the user terminal identifies the user as entering a butcher shop based on a street map (910) and the user's location on the street (920), the user terminal can identify the food category as meat. If the user terminal identifies the category as meat, the refrigerator can generate and provide shopping information related to meat to the user terminal.

[0128] FIG. 10 is a diagram illustrating obtaining store information based on map information to identify a category according to one embodiment of the present disclosure.

[0129] The user terminal can obtain the user's location information and map information. For example, the user terminal can obtain a street map (1010) and the user's location on the street (1020). The street map (1010) can include information related to the user's route, the location of stores, the entrances to each store, the size of each store, and the type of each store.

[0130] The user terminal can recognize that the user has entered a store based on location information and map information. For example, the user terminal can recognize that the user has entered the entrance of a store (1030) located in the upper left corner of the street map (1010) based on the street map (1010) and the user's location (1020) on the street.

[0131] The user terminal can use location and map information to identify whether the store the user has entered is a general store that carries multiple categories of food. For example, the user terminal can identify that the store (1030) the user has entered is a supermarket based on a street map (1010) and the user's location on the street (1020).

[0132] If the store the user has entered is a general store, the user terminal can provide shopping information regardless of the food category. If the store the user has entered is a general store, the user terminal can further identify the user's location within the store to identify the food category. For example, if the user terminal identifies the user's entry into a supermarket based on a street map (1010) and the user's location on the street (1020), the user terminal can receive shopping information related to the supermarket from the refrigerator. For example, if the user terminal identifies the user's entry into a supermarket based on a street map (1010) and the user's location on the street (1020), the user terminal can further identify the user's location within the supermarket to identify the food category.

[0133] FIG. 11 is a diagram illustrating obtaining shelf information based on indoor information to identify a category according to one embodiment of the present disclosure.

[0134] The user terminal can obtain the user's location information and indoor information within the store. For example, the user terminal can obtain an indoor map (1110) and the user's location (1130) within the store. The indoor map (1110) may include information related to the user's path, the location of a display shelf or corner, the size of the display shelf or corner, and the type of the display shelf or corner. The user terminal can obtain the user's location (1130) from at least one location measurement device (1121, 1122, 1123, 1124) placed within the store. Each of the at least one location measurement device (1121, 1122, 1123, 1124) may be a beacon that transmits a distinct and unique signal. The beacon may transmit a wireless communication signal such as Wi-Fi, UWB, or Bluetooth, an optical signal, or an acoustic signal such as an ultrasonic signal. The user terminal can determine the user's location (1130) by receiving the signal transmitted by the beacon.

[0135] The user terminal can identify shelf information related to a shelf or corner adjacent to the user based on indoor information. For example, the user terminal can recognize that the user is adjacent to the fresh food corner (1140) located in the upper left corner of the indoor map (1110) based on the indoor map (1110) and the user's indoor location (1130).

[0136] The user terminal can identify the food category based on shelf information. For example, if the user terminal recognizes that the user is near the fresh food corner (1140), the user terminal can identify the food category as vegetables. If the user terminal identifies the category as vegetables, the refrigerator can generate shopping information related to vegetables and provide it to the user terminal.

[0137] When a user moves through a supermarket, the user terminal can obtain the user's indoor location (1130) in real time. The user terminal can obtain shelf information corresponding to the user's location (1130) in real time. Based on the shelf information, the user terminal can identify food categories in real time. Accordingly, the refrigerator can update shopping information related to the identified food category in real time and provide it to the user terminal.

[0138] FIG. 12 is a diagram illustrating identification of a category based on a display image (1210) captured by a user terminal according to one embodiment of the present disclosure.

[0139] The user terminal may include a camera. The user terminal may use the camera to capture images of display shelves or corners within the store. The user terminal may capture images of the display shelves or corners within the store to obtain display images (1210). For example, the user terminal may capture a display image (1210) of the vegetable section within the store.

[0140] The user terminal can recognize information related to the category of food in the display image (1210). The user terminal can recognize an object related to the category of food in the display image (1210). The user terminal can recognize text (1220) related to the category of food in the display image (1210). For example, the user terminal can recognize the text "Vegetables" (1220) in the display image (1210). The user terminal can recognize a product image (1230) related to the category of food in the display image (1210). For example, the user terminal can recognize an image (1230) of vegetables such as tomatoes, carrots, and spinach in the display image (1210).

[0141] The user terminal can identify the category of food based on information related to the category of food recognized in the display image (1210). The user terminal can identify the category of food based on at least one of text (122) related to the category of food recognized in the display image (1210) and a product image (1230) related to the category of food. For example, if the user terminal recognizes the text (1220) "Vegetables" in the display image (1210), the user terminal can identify the category of food as vegetables. For example, if the user terminal recognizes the vegetable image (1230) in the display image (1210), the user terminal can identify the category of food as vegetables. If the user terminal identifies the category as vegetables, the refrigerator can generate shopping information related to vegetables and provide it to the user terminal.

[0142] The user terminal can identify the food category by recognizing the product code. The user terminal can use the camera to capture the product code, such as a QR code or barcode, on the food. The user terminal can obtain information about the food based on the captured product code. Based on the obtained information, the user terminal can identify the food category.

[0143] A user terminal can input the category of food for which shopping information is requested. The user terminal can display a UI for inputting shopping information. The user can select the category of food for which shopping information is requested through the UI. For example, the user can select the category of food for which shopping information is requested from among multiple categories displayed on the UI. The user can directly input the category of food for which shopping information is requested. For example, the user can directly input the category of food for which shopping information is requested via voice input.

[0144] FIG. 13 is a flowchart specifically illustrating a method for obtaining information related to food included in an identified category to provide shopping information according to one embodiment of the present disclosure.

[0145] In operation 1310, a system providing shopping information according to an embodiment of the present disclosure can obtain information on food stored in a home refrigerator. The home refrigerator or a server communicating with the home refrigerator can obtain food information, such as a list of stored foods and inventory records. The home refrigerator or the server communicating with the home refrigerator can store the obtained food information.

[0146] In operation 1320, a system for providing shopping information according to an embodiment of the present disclosure may obtain information related to food included in an identified category by communicating with a refrigerator or a server. A user terminal may communicate with the refrigerator or the server. The user terminal may request information related to food included in the identified category from among food information stored in the refrigerator or the server. The user terminal may receive information related to food included in the identified category from the refrigerator or the server. Based on the received information, the user terminal may output shopping information related to food included in the identified category. If the category of the food is not identified, the user terminal may output shopping information based on all food information stored in the refrigerator or the server.

[0147] FIG. 14a is a diagram illustrating outputting an internal image corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0148] In operation 1410, a user terminal (112) according to an embodiment of the present disclosure can identify the category of displayed food. The user terminal (112) can recognize that the user has entered the store. The user terminal (112) can determine the food category associated with the user's location.

[0149] In operation 1420, a user terminal (112) according to an embodiment of the present disclosure may transmit food category information to a refrigerator (121). In operation 1420, a user terminal (112) according to an embodiment of the present disclosure may transmit an internal image request to the refrigerator (121). The user terminal (112) may transmit food category information to the refrigerator (121). The user terminal (112) may request that an internal image of the refrigerator (121) be provided.

[0150] In operation 1430, a refrigerator (121) according to an embodiment of the present disclosure can identify whether food belonging to a category is stored. The refrigerator (121) can store a list of stored foods and an entry / exit record. The refrigerator (121) can receive food category information and a request from a user terminal (112). The refrigerator (121) that receives the food category information from the user terminal (112) can compare the food category information and the list of stored foods. The refrigerator (121) that receives the food category information from the user terminal (112) can compare the food category information and the entry / exit record. The refrigerator (121) can determine whether food belonging to the received food category is stored based on the list of stored foods and the entry / exit record.

[0151] In operation 1440, a refrigerator (121) according to an embodiment of the present disclosure can identify the storage location of food belonging to a category. The refrigerator (121) can recognize that food belonging to the corresponding food category is currently being stored in the refrigerator (121). When the refrigerator (121) recognizes that food belonging to the category is being stored, the refrigerator (121) can identify the location where the food is stored within the refrigerator (121) based on the entry / exit records or an internal preview image of the refrigerator (121).

[0152] In operation 1450, a refrigerator (121) according to an embodiment of the present disclosure can acquire an internal image including a storage location. The refrigerator (121) can capture an image of the location where the food is stored using an internal camera. The refrigerator (121) can control the camera for capturing the internal image to capture the internal image in real time.

[0153] If the refrigerator (121) includes multiple cameras that photograph each of multiple locations, the refrigerator (121) can selectively drive only the cameras that can photograph locations where food corresponding to a corresponding category is stored to capture an internal image. For example, if the refrigerator (121) includes internal cameras that can each photograph the upper compartment, the middle compartment, and the lower compartment of the storage room, if food corresponding to a corresponding category is in the upper compartment, the refrigerator (121) can drive only the camera that photographs the upper compartment to capture an internal image of the upper compartment.

[0154] If the refrigerator (121) cannot capture an internal image in real time, the refrigerator (121) can acquire the most recently captured and stored internal image. If there are multiple internal images captured at different internal locations within the refrigerator (121), the refrigerator (121) can only acquire an internal image captured at a location where food of the corresponding category is stored. For example, if the category is milk and milk is currently stored in a door bin, the door bin can be captured only when the refrigerator (121) door is opened, so the internal image cannot be captured in real time. In this case, the refrigerator (121) can acquire the last internal image captured at the door bin instead of the internal image captured in real time.

[0155] In operation 1460, a refrigerator (121) according to an embodiment of the present disclosure can correct an internal image. The refrigerator (121) can highlight and display food corresponding to a food category in the internal image. The refrigerator (121) can post-process the internal image to display information about the food corresponding to the food category in the internal image. For example, when the food category in the internal image is milk, the refrigerator (121) can display the location and information about milk in the internal image as a red rectangle.

[0156] In operation 1470, a refrigerator (121) according to an embodiment of the present disclosure can transmit location information and internal images. The refrigerator (121) can transmit internal images and stored food information to a user terminal (112).

[0157] If food belonging to the current category is not in the refrigerator (121), the refrigerator (121) can transmit information that there is no food belonging to the current category to the user terminal (112). If food belonging to the current category is not in the refrigerator (121), the refrigerator (121) can transmit an image of the entire interior of the refrigerator (121) to the user terminal (112). If food belonging to the current category is not in the refrigerator (121), the refrigerator (121) can transmit information related to food stored throughout the interior of the refrigerator (121) to the user terminal (112).

[0158] In operation 1480, a user terminal (112) according to an embodiment of the present disclosure can output location information and an internal image. The user terminal (112) can display the internal image received from the refrigerator (121) and information on stored food.

[0159] FIG. 14b is a diagram illustrating outputting an internal image corresponding to a category to provide shopping information according to one embodiment of the present disclosure.

[0160] In operation 1410, a user terminal (112) according to an embodiment of the present disclosure can identify the category of displayed food. The user terminal (112) can recognize that the user has entered the store. The user terminal (112) can determine the food category associated with the user's location.

[0161] In operation 1420, a user terminal (112) according to an embodiment of the present disclosure may transmit food category information to a server (130). In operation 1420, a user terminal (112) according to an embodiment of the present disclosure may transmit an internal image request to the server (130). The user terminal (112) may transmit food category information to the server (130). The user terminal (112) may transmit a signal requesting the server (130) to provide an internal image of the refrigerator (121).

[0162] In operation 1430, a server (130) according to an embodiment of the present disclosure can identify whether food belonging to a category is stored. The server (130) can receive a list of stored foods and an entry / exit record from a refrigerator (121). The server (130) can store the list of stored foods and the entry / exit records. The server (130) can receive food category information and a request from a user terminal (112). The server (130) that receives the food category information from the user terminal (112) can compare the food category information and the stored list of stored foods. The server (130) that receives the food category information from the user terminal (112) can compare the food category information and the entry / exit records. Based on the list of stored foods and the entry / exit records, the server (130) can determine whether food belonging to the received food category is stored in the refrigerator (121).

[0163] In operation 1440, a server (130) according to an embodiment of the present disclosure can identify the storage location of food belonging to a category. The server (130) can recognize that food belonging to the corresponding food category is currently being stored in the refrigerator (121). If the server (130) recognizes that food belonging to the category is being stored, the server can determine the location where the food is stored within the refrigerator (121) based on the entry / exit records or an internal preview image of the refrigerator (121).

[0164] In operation 1451, the server (130) according to an embodiment of the present disclosure can identify whether real-time shooting is possible. The server (130) can identify whether the refrigerator (121) can shoot the inside of the refrigerator (121) in real time. If the refrigerator (121) can shoot the inside of the refrigerator (121) in real time (operation 1451 - YES), the server (130) can request the refrigerator (121) to proceed to operation 1453. If the refrigerator (121) cannot shoot the inside of the refrigerator (121) in real time (operation 1451 - NO), the server (130) can proceed to operation 1455.

[0165] In operation 1453, a refrigerator (121) according to an embodiment of the present disclosure can acquire an internal image by operating a camera capable of photographing a storage location. The refrigerator (121) can receive a request from the server (130) to photograph an internal image in real time. The refrigerator (121) can photograph the location where the food is stored using an internal camera. The refrigerator (121) can control the camera for photographing the internal image to photograph the internal image in real time.

[0166] If the refrigerator (121) includes multiple cameras that photograph each of multiple locations, the refrigerator (121) can selectively drive only the cameras that can photograph locations where food corresponding to a corresponding category is stored to capture an internal image. For example, if the refrigerator (121) includes internal cameras that can each photograph the upper compartment, the middle compartment, and the lower compartment of the storage room, if food corresponding to a corresponding category is in the upper compartment, the refrigerator (121) can drive only the camera that photographs the upper compartment to capture an internal image of the upper compartment.

[0167] If the refrigerator (121) cannot capture an internal image in real time, the refrigerator (121) can acquire the most recently captured and stored internal image. If there are multiple internal images captured at different internal locations within the refrigerator (121), the refrigerator (121) can only acquire an internal image captured at a location where food of the corresponding category is stored. For example, if the category is milk and milk is currently stored in a door bin, the door bin can be captured only when the refrigerator (121) door is opened, so the internal image cannot be captured in real time. In this case, the refrigerator (121) can acquire the last internal image captured at the door bin instead of the internal image captured in real time.

[0168] In operation 1455, the server (130) according to one embodiment of the present disclosure may acquire a pre-stored internal image including a storage location. The server (130) may store an internal image previously received from the refrigerator (121). The server (130) may acquire a stored internal image when the refrigerator (121) is unable to capture the interior of the refrigerator (121) in real time.

[0169] In operation 1460, the server (130) according to one embodiment of the present disclosure may correct the internal image. The server (130) may highlight and display food corresponding to a food category in the internal image. The server (130) may post-process the internal image to display information about the food corresponding to the food category in the internal image. For example, when the food category in the internal image is milk, the server (130) may display the location and information about milk in the internal image as a red rectangle.

[0170] In operation 1470, a server (130) according to an embodiment of the present disclosure may transmit location information and internal images. The server (130) may transmit internal images and stored food information to a user terminal (112).

[0171] If food belonging to the current category is not in the refrigerator (121), the server (130) can transmit information that there is no food belonging to the current category to the user terminal (112). If food belonging to the current category is not in the refrigerator (121), the server (130) can transmit an image of the entire interior of the refrigerator (121) to the user terminal (112). If food belonging to the current category is not in the refrigerator (121), the server (130) can transmit information related to food stored throughout the interior of the refrigerator (121) to the user terminal (112).

[0172] In operation 1480, a user terminal (112) according to an embodiment of the present disclosure can output location information and an internal image. The user terminal (112) can display the internal image and stored food information received from the server (130).

[0173] FIG. 15 is a drawing showing an internal image (1500) corresponding to a category according to one embodiment of the present disclosure.

[0174] The refrigerator (121) can acquire an internal image (1500). The refrigerator (121) can acquire the current or most recently stored internal image (1500). The internal image (1500) can visually display food stored in the refrigerator (121). For example, the refrigerator (121) can visually display food stored in each compartment of the storage room inside the refrigerator (121).

[0175] The refrigerator (121) can highlight and display food corresponding to a food category in the internal image (1500). The refrigerator (121) can post-process the internal image to display information about food corresponding to the identified category in the internal image. For example, when the category in the internal image is milk, the refrigerator (121) can display information about the location and availability of milk in the internal image.

[0176] FIG. 16 is a diagram illustrating the output of storage list information and required list information corresponding to a category to provide shopping information according to one embodiment of the present disclosure. FIG. 16 illustrates a case where a refrigerator (121) generates the storage list information and required list information. However, this is not limited thereto, and a server (130) may also generate the storage list information and required list information.

[0177] In operation 1610, a user terminal (112) according to an embodiment of the present disclosure can identify the category of displayed food. The user terminal (112) can recognize that the user has entered the store. The user terminal (112) can determine the food category associated with the user's location.

[0178] In operation 1620, a user terminal (112) according to an embodiment of the present disclosure may request storage list information and required list information. The user terminal (112) may transmit food category information to a refrigerator (121) or a server (130). The user terminal (112) may request a list of required foods and foods to be purchased from the refrigerator (121) or the server (130).

[0179] In operation 1630, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may identify the inventory of food belonging to a category. The refrigerator (121) or the server (130) may receive a request from the user terminal (112) to provide a required list. In response to receiving the request, the refrigerator (121) or the server (130) may search for food belonging to the category from the stored food list and the receipt / delivery records. The refrigerator (121) or the server (130) may identify the inventory of food belonging to the corresponding category based on the stored food list and the receipt / delivery records.

[0180] In operation 1640, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may determine whether food belonging to a category needs to be purchased. The refrigerator (121) or the server (130) may determine whether food belonging to the identified category needs to be added to a need list. The refrigerator (121) or the server (130) may determine whether food needs to be purchased based on current inventory, expiration dates of food currently in storage, food entry and exit patterns, planned meal plan information, etc. The refrigerator (121) or the server (130) may also generate a need to purchase list by extracting only food belonging to the corresponding category from a need-to-purchase list of foods stored in the refrigerator (121) or the server (130).

[0181] In operation 1650, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may transmit storage list information and required list information. The refrigerator (121) or the server (130) may transmit a list of stored foods and a list of foods that need to be purchased in a corresponding category to a user terminal (112).

[0182] In operation 1660, a user terminal (112) according to an embodiment of the present disclosure can output storage list information and required list information. The user terminal (112) can display storage list information and required list information received from a refrigerator (121) or a server (130).

[0183] FIG. 17 is a diagram illustrating the output of recipe information corresponding to a category to provide shopping information according to one embodiment of the present disclosure. FIG. 17 illustrates a case where a refrigerator (121) generates recipe information. However, this is not limited to this case, and a server (130) may also generate recipe information.

[0184] In operation 1710, a user terminal (112) according to an embodiment of the present disclosure can identify the category of displayed food. The user terminal (112) can recognize that the user has entered the store. The user terminal (112) can determine the category of food associated with the user's location.

[0185] In operation 1720, a user terminal (112) according to an embodiment of the present disclosure may request recipe information. Once a category is determined, the user terminal (112) may transmit food category information to a refrigerator (121) or server (130) and request a recipe recommendation.

[0186] In operation 1730, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may determine a recommended recipe related to food belonging to a category. The refrigerator (121) or the server (130) may receive a request for recipe recommendation from a user terminal (112). In response to receiving the request, the refrigerator (121) or the server (130) may search a food storage list. The refrigerator (121) or the server (130) may determine a recommended recipe based on the food currently stored in the refrigerator (121) and recipes that can be made with the food belonging to the category.

[0187] In operation 1740, a refrigerator (121) or server (130) according to an embodiment of the present disclosure may determine food items that need to be purchased for a recommended recipe. The refrigerator (121) or server (130) may determine food items that are currently in short supply in the recommended recipe as food items that need to be purchased.

[0188] In operation 1750, a refrigerator (121) or server (130) according to an embodiment of the present disclosure may transmit recipe information. The recipe information may include a recommended recipe and information on food items that need to be purchased to make the recipe. The refrigerator (121) or server (130) may transmit the recommended recipe, the currently stored food items among the recipe ingredients, and the information on food items that need to be purchased to the user terminal (112).

[0189] In operation 1750, a user terminal (112) according to an embodiment of the present disclosure may output recipe information. The user terminal (112) may display the received recommended recipe and information on food products requiring purchase.

[0190] Figure 18 is a diagram illustrating outputting recommended storage locations for displayed food to provide shopping information according to one embodiment of the present disclosure. Figure 18 illustrates a case where a refrigerator (121) generates recommended storage locations. However, this is not limited to this case, and a server (130) may also generate recommended storage locations.

[0191] In operation 1810, a user terminal (112) according to an embodiment of the present disclosure can identify displayed food. The user terminal (112) can recognize the product code of the food. The user terminal (112) can capture an image of the food. The user terminal (112) can obtain food information based on the product code or the image of the food.

[0192] In operation 1820, a user terminal (112) according to an embodiment of the present disclosure may request recommended storage location information. The user terminal (112) may transmit food information to a refrigerator (121) or a server (130). The user terminal (112) may request the refrigerator (121) or the server (130) to provide recommended storage location information.

[0193] In operation 1830, a refrigerator (121) or server (130) according to an embodiment of the present disclosure may determine whether a purchase is necessary. The refrigerator (121) or server (130) may identify the inventory of recognized food based on a food storage list. The refrigerator (121) or server (130) may determine whether a purchase is necessary for the recognized food based on the food storage list.

[0194] In operation 1840, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure can identify a storage location. When a recognized food product needs to be purchased, the refrigerator (121) or the server (130) can identify whether there is a storage location in the refrigerator (121) or the server (130) for the recognized food product when purchased.

[0195] In operation 1850, a refrigerator (121) or server (130) according to an embodiment of the present disclosure may generate a guide indicating whether or not the food can be stored. If there is free storage space for the recognized food, the refrigerator (121) or server (130) may display a guide indicating that the recognized food can be stored. If there is no free storage space for the recognized food, the refrigerator (121) or server (130) may display a guide indicating that the recognized food cannot be stored.

[0196] In operation 1860, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may obtain an internal image including a storage location. If there is food in stock and no purchase is required, the refrigerator (121) or the server (130) may obtain inventory information and an internal image of a location where the food is currently stored. If there is no food in stock and there is free storage space for food, the refrigerator (121) or the server (130) may provide an internal image of a location where the food to be purchased can be stored. If there is no free storage space for food, the refrigerator (121) or the server (130) may provide information indicating that the food to be purchased cannot be stored. If there is no free storage space for food, the refrigerator (121) or the server (130) may provide an internal image indicating that the food cannot be stored or a full internal image. The refrigerator (121) or server (130) may provide an internal image showing alternative storage space for the food to be purchased when there is no food in stock and no free storage space for the food.

[0197] In operation 1870, a refrigerator (121) or a server (130) according to an embodiment of the present disclosure may transmit recommended storage location information. The refrigerator (121) or the server (130) may transmit an internal image indicating the recommended storage location to the user terminal (112).

[0198] In operation 1880, a user terminal (112) according to an embodiment of the present disclosure may output recommended storage location information. The user terminal (112) may display the received recommended storage location information.

[0199] FIG. 19 is a diagram illustrating a UI for providing shopping information according to one embodiment of the present disclosure when entering a store and the food category is not identified. For example, when a user enters a general store such as a supermarket, the food category is not identified, and thus the user terminal may display a UI similar to FIG. 19.

[0200] According to one embodiment, a user terminal (112) may display a first UI (1910) when entering a store and the category of food is not identified. The user terminal (112) may notify the user that the user has entered the store through the first UI (1910). The user terminal (112) may display guidance related to the store entered through the first UI (1910). For example, the user terminal (112) may display guidance such as "You are shopping at Shinseon Mart!" through the first UI (1910).

[0201] The user terminal (112) may display menus related to food information that can be obtained from a store entered through the first UI (1910). The user terminal (112) may display menus that require a user's selection to display shopping information through the first UI (1910). The user terminal (112) may display submenus for selecting food information unrelated to the food category in the entered store. For example, the user terminal (112) may display a stored food list menu (1911), a refrigerator internal image menu (1912), a shortage food list menu (1913), and a food recognition (1914) menu.

[0202] When the stored food list menu (1911) is selected, the user terminal (112) may display the stored food list UI (1920). The stored food list UI (1920) may display information on all categories of food stored in the refrigerator (121). The stored food list UI (1920) may include images of food stored in the refrigerator (121), types of food stored in the refrigerator (121), storage periods of food stored in the refrigerator (121), and storage locations of food stored in the refrigerator (121).

[0203] When selecting the refrigerator internal image menu (1912), the user terminal (112) can display an internal image UI (1930). The internal image UI (1930) can display an image of the entire interior of the refrigerator (121). The internal image UI (1930) can display an image of the interior of the refrigerator (121) that allows for an overall view of food stored in the refrigerator (121).

[0204] When selecting the "Insufficient Food List" menu (1913), the user terminal (112) may display the "Insufficient Food List" UI (1940). The "Insufficient Food List" UI (1940) may display information on all categories of food that are out of stock or in need of purchase. The "Insufficient Food List" UI (1940) may include images of food items that require purchase and the types of food items that require purchase.

[0205] When selecting food recognition (1914), the user terminal (112) may display a UI that can directly recognize food in the store. The user terminal (112) may identify the category of food by taking a picture of the food in the store or directly recognizing the product code on the food in the store.

[0206] FIG. 20 is a diagram illustrating a UI for providing shopping information according to one embodiment of the present disclosure when a food category has been identified upon entering a store. For example, when a user enters a store and is at a specific shelf or corner, the food category has been identified, and thus the user terminal may display a UI similar to FIG. 20.

[0207] According to one embodiment, a user terminal (112) may display a second UI (2010) upon entering a store and identifying a food category. The user terminal (112) may indicate that the food category has been identified in the second UI (2010). The user terminal (112) may indicate that the user is in a specific shelf or corner of the store in the second UI (2010). For example, the user terminal (112) may display a message such as "You are shopping in the vegetable section of a fresh market!" in the second UI (2010).

[0208] The user terminal (112) may display menus related to the categories identified in the second UI (2010). The user terminal (112) may display menus requesting a user's selection to display shopping information in the second UI (2010). The user terminal (112) may display submenus for selecting food information corresponding to the food category in the entered store. For example, the user terminal (112) may display a stored food list menu (2011), a refrigerator interior image menu (2012), a shortage food list menu (2013), and a recipe recommendation (2014) menu.

[0209] When the stored food list menu (2011) is selected, the user terminal (112) may display the stored food list UI (2020). The stored food list UI (2020) may display information on an identified category of food stored in the refrigerator (121). For example, if the identified category is vegetables, the stored food list UI (1920) may include an image of the food stored in the refrigerator (121), the type of food stored in the refrigerator (121), the storage period of the food stored in the refrigerator (121), and the storage location of the food stored in the refrigerator (121).

[0210] When selecting the refrigerator internal image menu (2012), the user terminal (112) may display an internal image UI (2030). The internal image UI (2030) may display an internal image captured at a location in the refrigerator (121) where food corresponding to a category is stored. For example, if the identified category is vegetables, the internal image UI (2030) may display an internal image captured at a location in the refrigerator (121) where vegetables are stored.

[0211] The internal image UI (2030) may include information on the arrival of food corresponding to a category. For example, if the identified category is vegetables, the internal image UI (2030) may include text indicating that one paprika was delivered to the refrigerator (121) two days ago. For example, if the identified category is vegetables, the user terminal (112) may highlight the location where the paprika was delivered with a rectangle on the internal image UI (2030).

[0212] When the "Insufficient Food List" menu (2013) is selected, the user terminal (112) may display the "Insufficient Food List" UI (2040). The "Insufficient Food List" UI (2040) may display information about foods that are out of stock or require purchase in the identified category. For example, if the identified category is vegetables, the user terminal (112) may determine that the food that requires purchase is carrots and display an image of a carrot and the fact that the food that requires purchase is carrots on the "Insufficient Food List" UI (2040).

[0213] When selecting a recipe recommendation (2014), the user terminal (112) may display a recipe recommendation UI (2050). The user terminal (112) may display a recommended recipe related to the food to be purchased on the recipe recommendation UI (2050). For example, if the food to be purchased is carrots, the user terminal (112) may display a recipe recommendation UI (2050) recommending a recipe for carrot pancakes that can be made using carrots.

[0214] Figure 21 is a diagram illustrating a UI for providing shopping information according to one embodiment of the present disclosure when a food category is identified based on user input. For example, if a user directly recognizes the product code of a specific food, the food category is individually recognized, and thus the user terminal can display a UI similar to Figure 21.

[0215] According to one embodiment, the user terminal (112) may display a third UI (2110) when the category of food is to be identified based on a user input. The user terminal (112) may request to take a picture to identify the category of food in the third UI (2110). The user terminal (112) may guide the user to take a picture of information related to the food in the third UI (2110) to identify the category of food. For example, the user terminal (112) may display a guide saying, "Please take a picture of the inside of the store, a picture of the product, or a product code" in the third UI (2110). When the user terminal (112) recognizes that the user has entered the store, the third UI (2110) may guide the user to take a picture.

[0216] The user terminal (112) may request the user to identify the category of food directly when the user selects it in the first UI (1910) after entering the store. The user terminal (112) may request the user to identify the category when the user fails to automatically recognize the category after entering the store.

[0217] The user terminal (112) can receive input other than image recognition by camera shooting through the third UI (2110). For example, the user terminal (112) can guide voice input through the third UI (2110). For example, the user terminal (112) can guide text input through the third UI (2110). For example, the user terminal (112) can receive an input for selecting a desired category for which shopping information is to be provided among preset categories through the third UI (2110).

[0218] According to one embodiment, the user terminal (112) may display a food image (2120) if the user takes a picture of the food image after displaying the third UI (2110). The user terminal (112) may display the food taken in the food image (2120).

[0219] The user terminal (112) can extract objects related to the category of food from the food image (2120). The user terminal (112) can extract text related to the category of food from the food image (2120). For example, the user terminal (112) can extract the text (2121) titled "Vegetables" from the food image (2120). For example, the user terminal (112) can highlight (2122) the vegetable image in the food image (2120). The user terminal (112) can display the category of food identified in the food image (2120). For example, the user terminal (112) can display a message saying "The vegetable corner has been recognized" in the food image (2120).

[0220] A user can select a confirmation menu to approve a category identified in the food image (2120) to display a menu related to the identified category. If the user wishes to display a menu related to a category other than the one identified in the food image (2120), the user can select the recapture menu and re-capture the food image.

[0221] The user terminal (112) may display a third UI (2130) when it wishes to display shopping information related to an identified category. The user terminal (112) may display menus related to the identified category in the third UI (2130). The user terminal (112) may display menus requesting a user's selection to display shopping information in the third UI (2130). The user terminal (112) may display submenus for selecting food information corresponding to a food category in the entered store. For example, the user terminal (112) may display a stored food list menu (2131), a refrigerator interior image menu (2132), a shortage food list menu (2133), and a recipe recommendation (2134) menu.

[0222] FIG. 22 is a diagram illustrating a UI when individual foods are identified based on user input to provide shopping information according to one embodiment of the present disclosure.

[0223] According to one embodiment, the user terminal (112) may display the fourth UI (2210) when the category of food is to be identified based on a user input. The user terminal (112) may request to take a picture to identify the category of food in the fourth UI (2210). The user terminal (112) may guide the user to take a picture of food-related information to identify the category of food in the fourth UI (2210). For example, the user terminal (112) may display a guide saying, "Please take a picture of the inside of the store, a picture of the product, or a product code" in the fourth UI (2210). When the user terminal (112) recognizes that the user has entered the store, the fourth UI (2210) may guide the user to take a picture.

[0224] According to one embodiment, the user terminal (112) may display a product code screen (2220) after displaying the fourth UI (2210) if the user takes a picture of a food code. The user terminal (112) may extract food information from the product code screen (2220). The user terminal (112) may display the food information extracted from the product code screen (2220). For example, the user terminal (112) may display the food information "Seoul Milk 1L has been recognized" on the product code screen (2220).

[0225] The user terminal (112) may request the refrigerator (121) to transmit information about the identified food in response to the user selecting a confirmation menu for approving the food identified on the product code screen (2220). If the user terminal (112) wishes to recognize a product other than the food identified on the product code screen (2220), the user terminal (112) may select the shooting menu again and obtain the product code again.

[0226] The refrigerator can check the inventory information of the food identified in the food image (2120) and whether there is a storage location in the refrigerator for the identified food when purchased. The refrigerator can transmit the information obtained as a result of the check to the user terminal (112).

[0227] The user terminal (112) may display an inventory information UI (2230) when the identified food is in stock in the refrigerator. The user terminal (112) may display a message indicating that the identified food remains in stock in the inventory information UI (2230). For example, the user terminal (112) may display a message indicating that "there is currently milk left in the refrigerator" in the inventory information UI (2030). The user terminal (112) may display an internal image of the refrigerator showing the inventory of the identified food in the inventory information UI (2030). The user terminal (112) may display information related to the inventory, such as the type of the inventory food and the date of receipt of the inventory food, in the internal image. The user terminal (112) may highlight the inventory food in the internal image. For example, the user terminal (112) may highlight milk with a rectangular box in the internal image.

[0228] When the identified food is purchased, the user terminal (112) may display a storage location guidance UI (2240) if there is a storage location in the refrigerator. When the identified food is purchased in the storage location guidance UI (2240), the user terminal (112) may display an internal image indicating a storage location inside the refrigerator. For example, when milk is purchased in the storage location guidance UI (2240), the user terminal (112) may display an internal image indicating a door bin in which milk can be stored. When the identified food is purchased in the storage location guidance UI (2240), the user terminal (112) may guide a storage location inside the refrigerator. For example, the user terminal (112) may display a guidance phrase saying, "If you purchase milk, you can store it here" in the storage location guidance UI (2240).

[0229] When the identified food is purchased and there is no storage location in the refrigerator, the user terminal (112) may display a UI (2250) indicating that the food cannot be stored. The user terminal (112) may provide guidance in the UI (2250) indicating that there is no storage location in the refrigerator when the identified food is purchased. For example, the user terminal (112) may display a guidance phrase such as "Even if you purchase milk, there is no storage location in the refrigerator." Instead of the UI (2250) indicating that the food cannot be stored, the user terminal (112) may display a UI indicating an alternative storage location when the identified food is purchased.

[0230] A method for providing shopping information according to the present disclosure includes an operation of identifying a category of food sold in a store, an operation of generating shopping information related to stored food included in the identified category among at least one food stored in a refrigerator at home, and an operation of transmitting the generated shopping information to a user terminal, wherein the shopping information may include at least one of a list of stored food, a location of stored food, an internal image of a refrigerator displaying stored food, a list of items that need to be purchased in the category, recipe information related to sold food and stored food, and recommended storage location information related to where to store sold food in the refrigerator.

[0231] The method for providing shopping information according to the present disclosure may include an operation in which a user terminal identifies that a user has entered a store using location information and map information.

[0232] A method for providing shopping information according to the present disclosure may include an operation of obtaining an image of the interior of the refrigerator by photographing the interior of the refrigerator and at least one food item being brought in and out of the refrigerator through at least one camera.

[0233] An operation of identifying a category according to the present disclosure may include an operation of a user terminal acquiring at least one piece of information from among user location information, information related to displayed food, and information related to a display shelf on which the displayed food is placed, and an operation of identifying a category based on the acquired at least one piece of information.

[0234] The operation of generating shopping information according to the present disclosure may include an operation of a user terminal requesting an internal image corresponding to a category from a refrigerator, an operation of identifying whether stored food is stored and a storage location, an operation of obtaining an internal image including the storage location, and an operation of transmitting the internal image to the user terminal.

[0235] A method for providing shopping information according to the present disclosure may include, when there is no stored food corresponding to a category in a refrigerator, an operation of transmitting information indicating that there is no stored food and an entire internal image of the refrigerator to a user terminal.

[0236] The operation of generating shopping information according to the present disclosure may include an operation of a user terminal requesting a refrigerator for a list of items that need to be purchased corresponding to a category, an operation of determining whether or not food for sale needs to be purchased, and an operation of transmitting the list of items to the user terminal.

[0237] The operation of generating shopping information according to the present disclosure may include an operation of a user terminal requesting recipe information corresponding to a category from a refrigerator, an operation of determining recommended recipes related to stored food and sold food, and an operation of transmitting the recipe information to the user terminal.

[0238] The operation of generating shopping information according to the present disclosure may include an operation of a user terminal requesting a refrigerator for recommended storage location information corresponding to a category, an operation of identifying whether the food for sale can be stored in a recommended storage location corresponding to the category, and an operation of transmitting recommended storage location information including the recommended storage location to the user terminal.

[0239] The method for providing shopping information according to the present disclosure may include an operation of transmitting to a user terminal, when a food product for sale cannot be stored in a recommended storage location in a refrigerator, information indicating that the food product for sale cannot be stored or information on an alternative storage location where the food product for sale can be stored other than the recommended storage location.

[0240] A refrigerator applying a shopping information providing method according to the present disclosure includes a communication module (190), at least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343), and at least one processor, wherein the at least one processor receives a result of identifying a category of food sold in a store from a user terminal (112), generates shopping information related to stored food included in the identified category among at least one stored food, and transmits the generated shopping information to the user terminal (112), and the shopping information may include at least one of a list of stored food, a location of auxiliary food, an internal image displaying stored food, a list of items that need to be purchased in a category, recipe information related to sold food and stored food, and recommended storage location information related to where to store sold food.

[0241] At least one processor according to the present disclosure can receive a result identifying that a user has entered the store using location information and map information.

[0242] At least one processor according to the present disclosure can obtain an image of the inside of the refrigerator by photographing the inside of the refrigerator and at least one food item being put in and out of the refrigerator through at least one camera.

[0243] At least one processor according to the present disclosure can receive at least one piece of information acquired from a user terminal among user location information, information related to displayed food, and information related to a display shelf on which displayed food is placed, and identify a category based on the acquired at least one piece of information.

[0244] At least one processor according to the present disclosure may receive a signal requesting an internal image corresponding to a category from a user terminal, identify whether stored food is stored and the storage location, obtain an internal image including the storage location, and transmit the internal image to the user terminal.

[0245] At least one processor according to the present disclosure can transmit information that there is no stored food corresponding to a category in the refrigerator and an entire internal image of the refrigerator to the user terminal.

[0246] At least one processor according to the present disclosure can receive a signal requesting a list of items that need to be purchased corresponding to a category from a user terminal, determine whether the items for sale need to be purchased, and transmit the list of items to the user terminal.

[0247] At least one processor according to the present disclosure may receive a signal requesting recipe information corresponding to a category from a user terminal, determine a recommended recipe related to stored food and sold food, and transmit the recipe information to the user terminal.

[0248] At least one processor according to the present disclosure may receive a signal requesting recommended storage location information corresponding to a category from a user terminal, identify whether the food for sale can be stored in a recommended storage location corresponding to the category, and transmit recommended storage location information including the recommended storage location to the user terminal.

[0249] At least one processor according to the present disclosure may transmit to the user terminal, when the food for sale cannot be stored in the recommended storage location in the refrigerator, information indicating that the food for sale cannot be stored or information on an alternative storage location where the food for sale can be stored other than the recommended storage location.

[0250] The method for providing shopping information according to the present disclosure and the refrigerator applying the method automatically provide information on food currently stored or food to be purchased to a user who wishes to shop for food without any separate operation, thereby preventing duplicate purchases or omissions in purchases.

[0251] In addition, the method for providing shopping information according to the present disclosure and the refrigerator applying the method can efficiently provide shopping information according to the user's movement path by determining the food category related to the store or shelf where the user is currently located and providing customized information related to the food category.

[0252] A method according to an embodiment of the present disclosure may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present disclosure or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0253] Some embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media. Furthermore, some embodiments of the present disclosure may also be implemented as a computer program or computer program product containing computer-executable instructions, such as a computer program that is executed by a computer.

[0254] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0255] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. An action to identify the category of food sold in a store; An action of generating shopping information related to stored food belonging to the identified category among at least one food stored in a refrigerator in a home; and Including an action of transmitting the generated shopping information to a user terminal, A method for providing shopping information, wherein the shopping information includes at least one of a list of the stored food, a location of the stored food, an internal image of the refrigerator displaying the stored food, a list of items that need to be purchased in the category, recipe information related to the sold food and the stored food, and recommended storage location information related to where to store the sold food in the refrigerator.

2. In paragraph 1, A method further comprising: the user terminal using location information and map information to identify that the user has entered the store.

3. In at least one of paragraphs 1 and 2, A method further comprising: obtaining an image of the inside of the refrigerator by photographing the inside of the refrigerator and the at least one food item being put into or taken out of the refrigerator through at least one camera.

4. In at least one of clauses 1 to 3, The action of identifying the above categories is; An operation in which the user terminal acquires at least one of the user's location information, information related to the displayed food, and information related to the display shelf on which the displayed food is placed; and A method comprising an operation of identifying the category based on at least one piece of information acquired above.

5. In at least one of clauses 3 to 4, The action of generating the above shopping information is: An action of the user terminal requesting the internal image corresponding to the category; An action to identify whether the above stored food is stored and where it is stored; An operation of obtaining said internal image including said storage location; and A method comprising an action of transmitting the internal image to the user terminal.

6. In paragraph 5, A method further comprising: transmitting, to the user terminal, information that the stored food does not exist and an entire internal image of the refrigerator, if the stored food corresponding to the category does not exist in the refrigerator.

7. In at least one of clauses 1 to 4, The action of generating the above shopping information is: An action of the user terminal requesting the refrigerator for the list of items requiring purchase corresponding to the category; An action to determine whether or not the above food for sale needs to be purchased; and A method comprising the action of transmitting the above-mentioned necessary list to the user terminal.

8. In at least one of clauses 1 to 4, The action of generating the above shopping information is: An action of the user terminal requesting the refrigerator for the recipe information corresponding to the category; An operation of determining a recommended recipe related to said stored food and said sold food; and A method comprising an action of transmitting the above recipe information to the user terminal.

9. In at least one of clauses 1 to 4, The action of generating the above shopping information is: An action of the user terminal requesting the refrigerator for the recommended storage location information corresponding to the category; An action to identify whether the above-mentioned food for sale can be stored in a recommended storage location corresponding to the above-mentioned category; and A method comprising the action of transmitting the recommended storage location information including the recommended storage location to the user terminal.

10. In paragraph 9, A method further comprising an action of transmitting to the user terminal, when the food for sale cannot be stored in the recommended storage location in the refrigerator, information indicating that the food for sale cannot be stored or information on an alternative storage location where the food for sale can be stored other than the recommended storage location.

11. Communication module (190); At least one camera (321, 322, 323, 331, 332, 333, 341, 342, 343); and comprising at least one processor, At least one processor of the above, The result of identifying the category of food sold in the store is received from the user terminal (112), Generate shopping information related to stored food included in the above-identified category among at least one stored food, and The generated shopping information is transmitted to the user terminal (112), A refrigerator (121), wherein the shopping information includes at least one of a list of the stored food, a location of the stored food, an internal image displaying the stored food, a list of items that need to be purchased in the category, recipe information related to the sold food and the stored food, and recommended storage location information related to where to store the sold food.

12. In paragraph 11, At least one processor of the above, A refrigerator (121) in which the user terminal receives a result identifying that the user has entered the store using location information and map information.

13. In at least one of paragraphs 11 and 12, At least one processor of the above, A refrigerator (121) that obtains an image of the inside of the refrigerator by photographing the inside of the refrigerator and at least one food item being put in and taken out of the refrigerator through at least one camera.

14. In at least one of clauses 11 to 13, At least one processor of the above, The user terminal receives at least one piece of information obtained from the user's location information, information related to the displayed food, and information related to the display shelf on which the displayed food is placed. A refrigerator (121) that identifies the category based on at least one piece of information acquired above.

15. In at least one of clauses 13 to 14, At least one processor of the above, Receive a signal requesting the internal image corresponding to the above category from the above user terminal, Identify whether the above stored food is stored and where it is stored, Obtaining the internal image including the storage location, and A refrigerator (121) that transmits the internal image to the user terminal.

Images