Electronic device and control method thereof

Abstract

This electronic device for storing wine comprises a door, a memory storing instructions, a camera, and at least one processor, wherein the instructions, when executed by the at least one processor, cause the electronic device to: when the door is opened, acquire a captured image through the camera; on the basis of the captured image, acquire wine storage information including a location of a shelf on which wine is placed and a location of the wine; acquire wine characteristics information corresponding to the wine; and store, in the memory, wine information including the wine storage information and the wine characteristics information.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof for storing a target object and providing information related to the storage.

[0002] A device that stores a target object (e.g., wine) can perform storage-related functions (e.g., temperature maintenance). The device may include a shelf or container for storing the target object. If the user were to manually input the location on the shelf where the target object is stored, this could be inconvenient for the user.

[0003] If the user takes a picture of the target object and registers it on the storage server, it may cause inconvenience to the user as the user has to take a picture of the wine himself.

[0004] If you use a terminal device to take pictures every time you store wine, you may experience inconveniences such as having to find the storage location, launch the wine application, or check for an internet connection.

[0005] Additionally, in the process of users directly taking pictures of wine, there was a possibility that the wine recognition rate would drop due to differences in the shooting conditions (distance between the subject and the lens, lighting, etc.).

[0006] Additionally, when trying to store a large number of wines at once, there was the inconvenience of having to register a large number of wines one by one.

[0007] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof that obtains a photographic image including both a shelf and wine using a camera included in the electronic device, and automatically specifies and stores the shelf position and wine position.

[0008] According to one embodiment, an electronic device for storing wine includes a door, a memory storing instructions, a camera, and at least one processor, wherein the instructions, when executed by the at least one processor, cause the electronic device to, when the door is opened, obtain a photographed image through the camera, obtain wine storage information including a shelf position on which the wine is placed and a wine position based on the photographed image, obtain wine characteristic information corresponding to the wine, and store wine information including the wine storage information and the wine characteristic information in the memory.

[0009] The electronic device further includes a door sensor, and the instructions, when executed by the at least one processor, can cause the electronic device to obtain the photographed image through the camera when a signal indicating that the door is opened is received from the door sensor.

[0010] The electronic device further includes a plurality of shelves, and the instructions, when executed by the at least one processor, cause the electronic device to identify a shelf object in the photographed image, obtain a horizontal length of the shelf object, and obtain a shelf position based on the horizontal length of the shelf object and a first mapping table stored in the memory, wherein the first mapping table may include a plurality of horizontal length ranges corresponding to each of the plurality of shelves.

[0011] The above instructions, when executed by the at least one processor, may cause the electronic device to obtain first coordinate information corresponding to the shelf object based on the photographed image, and to obtain the length of the outermost line of the shelf object in the lowermost direction among a plurality of outer lines included in the photographed image as the horizontal length of the shelf object based on the first coordinate information.

[0012] The instructions, when executed by the at least one processor, may cause the electronic device to identify a plurality of vertices corresponding to the shelf object based on the first coordinate information, identify two vertices adjacent to the camera among the plurality of vertices, and obtain the length of the outline connecting the two vertices as the horizontal length of the shelf object.

[0013] The instructions, when executed by the at least one processor, may cause the electronic device to identify a horizontal length range corresponding to the horizontal length among the plurality of horizontal length ranges, and to identify a first shelf corresponding to the identified horizontal length range as a shelf where the wine is stored.

[0014] The electronic device wherein the instructions, when executed by the at least one processor, identify the shelf object based on the horizontal length of the shelf object, and the shelf object is determined to be at a greater distance from the camera as the horizontal length of the shelf object decreases.

[0015] The instructions, when executed by the at least one processor, cause the electronic device to identify a wine object in the photographed image, obtain relative position information indicating a position of the wine object in an area of ​​the first shelf based on the photographed image, and obtain the wine position based on the relative position information and a second mapping table stored in the memory, wherein the second mapping table may include a plurality of relative position ranges corresponding to each of a plurality of areas of the plurality of shelves.

[0016] The instructions, when executed by the at least one processor, may cause the electronic device to obtain first coordinate information corresponding to the shelf object based on the photographed image, obtain the length of a downward outline of the shelf object as the horizontal length of the shelf object based on the first coordinate information, obtain a first x-axis minimum value of the shelf object based on the first coordinate information, obtain second coordinate information corresponding to the wine object based on the photographed image, obtain at least one of a second x-axis minimum value of the wine object or an x-axis maximum value of the wine object based on the second coordinate information, obtain the relative position information corresponding to the wine object based on at least one of the second x-axis minimum value of the wine object or the x-axis maximum value of the wine object, the first x-axis minimum value of the shelf object, and the horizontal length of the shelf object, and obtain the wine position based on the relative position information.

[0017] The instructions, when executed by the at least one processor, cause the electronic device to store a first wine sample image in a database through a wine registration process, and, when a wine object is identified in the photographed image, obtain the first wine sample image corresponding to the wine object, and obtain the wine characteristic information based on the first wine sample image, wherein the wine characteristic information may include at least one of a name, a type, a flavor, a manufacturer, a manufacturing date, an expiration date, a price, or a rating for the wine.

[0018] According to one embodiment, a method for controlling an electronic device for storing wine includes the steps of: acquiring a photographed image when a door of the electronic device is opened; acquiring wine storage information including a shelf position on which the wine is placed and a wine position based on the photographed image; acquiring wine characteristic information corresponding to the wine; and storing wine information including the wine storage information and the wine characteristic information.

[0019] The step of acquiring the above-described photographed image can acquire the above-described photographed image when a signal indicating that the door is opened is received.

[0020] The step of obtaining the wine storage information may include identifying a shelf object in the photographed image, obtaining a horizontal length of the shelf object, and obtaining the shelf position based on the horizontal length of the shelf object and a first mapping table stored in the electronic device, wherein the first mapping table may include a plurality of horizontal length ranges corresponding to each of a plurality of shelves of the electronic device.

[0021] The step of obtaining the above wine storage information may include obtaining first coordinate information corresponding to the shelf object based on the photographed image, and obtaining the length of the outer line in the lowest direction of the shelf object as the horizontal length of the shelf object based on the first coordinate information.

[0022] The step of obtaining the above wine storage information may include identifying a plurality of vertices corresponding to the shelf object based on the first coordinate information, identifying two vertices adjacent to the camera of the electronic device among the plurality of vertices, and obtaining the length of the outline connecting the two vertices as the horizontal length of the shelf object.

[0023] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0024] FIG. 1 is a drawing for explaining an electronic device for storing wine, according to one embodiment.

[0025] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0026] FIG. 3 is a block diagram illustrating the configuration of the electronic device of FIG. 2, according to one embodiment.

[0027] FIG. 4 is a diagram for explaining an operation of storing wine information according to one embodiment.

[0028] FIG. 5 is a diagram for explaining an operation of obtaining wine storage information and wine characteristic information according to one embodiment.

[0029] FIG. 6 is a diagram for explaining an operation of obtaining wine storage information according to one embodiment.

[0030] FIG. 7 is a drawing for explaining an operation of obtaining a shelf position according to one embodiment.

[0031] FIG. 8 is a diagram for explaining an operation of obtaining a wine position according to one embodiment.

[0032] FIG. 9 is a drawing for explaining the structure of a shelf included in an electronic device according to one embodiment.

[0033] FIG. 10 is a drawing for explaining an operation of identifying a shelf object through an image, according to one embodiment.

[0034] FIG. 11 is a drawing for explaining an operation of identifying a shelf position through an image, according to one embodiment.

[0035] Figure 12 is a drawing for explaining the point of identifying a shelf object.

[0036] Figure 13 is a drawing for explaining a table including information for specifying shelf positions.

[0037] FIG. 14 is a diagram illustrating an operation of identifying a wine location through an image, according to one embodiment.

[0038] FIG. 15 is a diagram illustrating an operation of identifying a wine object through an image, according to one embodiment.

[0039] FIG. 16 is a diagram illustrating a table including information for specifying a wine location, according to one embodiment.

[0040] FIG. 17 is a drawing for explaining an operation of cropping an image according to one embodiment.

[0041] FIG. 18 is a drawing for explaining an operation of identifying multiple wines according to one embodiment.

[0042] FIG. 19 is a diagram for explaining an operation of obtaining wine characteristic information through a wine server according to one embodiment.

[0043] FIG. 20 is a drawing for explaining an operation of displaying wine information according to one embodiment.

[0044] FIG. 21 is a drawing for explaining an operation of indicating a wine position according to one embodiment.

[0045] FIG. 22 is a drawing for explaining a situation in which multiple shelves are included in one image, according to one embodiment.

[0046] FIG. 23 is a drawing for explaining an operation of identifying whether wine is opened, according to one embodiment.

[0047] FIG. 24 is a diagram for explaining an image transformation operation for an area including a wine object, according to one embodiment.

[0048] FIG. 25 is a drawing for explaining an operation of registering a wine image in an electronic device according to one embodiment.

[0049] FIG. 26 is a diagram for explaining an operation of obtaining wine characteristic information using a registered wine image according to one embodiment.

[0050] FIG. 27 is a diagram for explaining a wine image database according to one embodiment.

[0051] FIG. 28 is a diagram for explaining an operation of obtaining wine characteristic information using a server according to one embodiment.

[0052] FIG. 29 is a diagram for explaining an operation of providing wine information according to one embodiment.

[0053] FIG. 30 is a diagram for explaining an operation of a terminal device providing wine information according to one embodiment.

[0054] FIG. 31 is a drawing for explaining an operation of providing a UI for inputting wine characteristic information according to one embodiment.

[0055] FIG. 32 is a drawing for explaining an operation of a terminal device providing a UI for inputting wine characteristic information, according to one embodiment.

[0056] FIG. 33 is a drawing for explaining a UI for entering wine characteristic information according to one embodiment.

[0057] FIG. 34 is a drawing for explaining a UI for entering wine characteristic information according to one embodiment.

[0058] FIG. 35 is a drawing for explaining a method for controlling an electronic device according to one embodiment.

[0059] The embodiments described in this disclosure and the configurations shown in the drawings are merely examples of embodiments, and various modifications may be made without departing from the scope and spirit of this disclosure.

[0060] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0061] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0062] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0063] The term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0064] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0065] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0066] The terms “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0067] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0068] When we say that a component is “on” another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0069] A refrigerator according to one embodiment may include a body.

[0070] The “body” may include an inner case, an outer case placed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0071] The "inner case" may include at least one of a case, plate, panel, or liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0072] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0073] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0074] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0075] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0076] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a “refrigerator,” a “freezer,” or a “variable temperature room,” which are distinguished by their intended use and / or temperature range. The refrigerator may be maintained at a temperature appropriate for refrigerating items, and the freezer may be maintained at a temperature appropriate for freezing items. “Refrigeration” may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. “Freezing” may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. The variable temperature room may be used as either a refrigerator or a freezer, at the user’s option or not.

[0077] In addition to being called “refrigerator,” “freezer,” and “variable temperature room,” a storage room can also be called by various other names, such as “vegetable room,” “fresh room,” “cooling room,” and “ice room.” The terms “refrigerator,” “freezer,” and “variable temperature room” used hereinafter should be understood to encompass storage rooms with corresponding uses and temperature ranges.

[0078] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0079] The “door” may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0080] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0081] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door basket for storing items.

[0082] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0083] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0084] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0085] A “cold air supply device” may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0086] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment by generating heat and cooling through the Peltier effect.

[0087] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0088] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be connected to the exterior of the main body to dissipate heat from components placed within the machine room.

[0089] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0090] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0091] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0092] The “control unit” may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0093] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0094] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals for controlling the operation of the cooling (air conditioning) supply unit. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling unit based on the temperature information.

[0095] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0096] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0097] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0098] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) to which the refrigerator or user device is connected to the wide area network (WAN) to which the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0099] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0100] The output interface may include a display, a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0101] Hereinafter, refrigerators according to various embodiments will be specifically described with reference to the attached drawings.

[0102] FIG. 1 is a drawing for explaining an electronic device (100) for storing wine, according to one embodiment.

[0103] The electronic device (100) may be a device that performs a refrigeration function. The electronic device (100) may be a refrigerator. The electronic device (100) may store a specific object. The electronic device (100) may perform a refrigeration function to maintain the temperature of the specific object.

[0104] The electronic device (100) may include a door (101). The door (101) may be opened or closed by a physical force of a user or an electrical force included in the electronic device (100). The electronic device (100) may include at least one shelf (10). A description of the plurality of shelves is provided in FIG. 9.

[0105] According to one embodiment, the electronic device (100) may be a wine refrigerator or wine cellar for storing wine.

[0106] The object stored in the electronic device (100) is described as wine. However, according to various embodiments, another object may be stored inside the electronic device (100) instead of wine.

[0107] The electronic device (100) may include a camera (190). The electronic device (100) may capture a photograph of an object (e.g., wine) stored in the electronic device (100) through the camera (190). The electronic device (100) may obtain a captured image (1) through the camera (190).

[0108] The electronic device (100) can acquire a photographed image (1) corresponding to a preset photographing direction based on the arrangement angle of the camera (190), etc. The photographed image (1) can include at least one of a shelf or a wine glass. The camera (190) can be placed on the upper portion of the front of the electronic device (100).

[0109] The electronic device (100) can obtain wine information based on the captured image (1). The wine information can include information related to wine stored in the electronic device (100).

[0110] FIG. 2 is a block diagram illustrating an electronic device (100) according to one embodiment.

[0111] Referring to FIG. 2, the electronic device (100) may include a door (101), a memory (110), a camera (190), and at least one processor (120).

[0112] The door (101) can be connected to the main body by an electronic device (100). The power status of the camera (190) can change based on whether the door (101) is open.

[0113] The memory (110) can store wine information obtained from the electronic device (100).

[0114] The camera (190) may be an image sensor that performs a photographing function. The camera (190) can acquire a photographed image. The camera (190) may be positioned at a preset angle to acquire a photographed image including a shelf or wine.

[0115] The electronic device (100) may be a storage or preservation device for storing a target object. The electronic device (100) may be a device that performs a refrigeration function to store the target object within a preset temperature range. The electronic device (100) may be a refrigerator. The target object may be in a liquid state, such as a beverage or alcoholic beverage. According to various embodiments, the target object may be food.

[0116] In one embodiment, the target object is wine, and the electronic device (100) may be a wine refrigerator. In various embodiments, objects other than wine may be stored. In the description below, wine may be described as the target object. The electronic device (100) may also be described as a storage device for the target object, rather than a wine refrigerator.

[0117] At least one processor (120) can perform overall control operations of the electronic device (100). At least one processor (120) can perform a function of controlling overall operations of the electronic device (100).

[0118] At least one processor (120) can obtain a photographed image through a camera (190) when the door (101) is opened, obtain wine storage information including the shelf position on which the wine is placed and the wine position based on the photographed image, obtain wine characteristic information corresponding to the wine, and store the wine information including the wine storage information and the wine characteristic information in the memory (110).

[0119] At least one processor (120) may transmit wine information to a user terminal, so that the user terminal may display at least one of wine storage information, wine characteristic information, storage recommendation based on the wine information, or food pairing recommendation based on the wine information.

[0120] At least one processor (120) can identify whether the door (101) is open. If the door (101) is opened, the at least one processor (120) can acquire a photographed image through the camera (190). The photographed image may be an RGB image. If the door (101) is not opened, the at least one processor (120) can keep the camera (190) in an off state or a power-saving state. The at least one processor (120) can acquire a photographed image only when an event of the door (101) being opened is identified.

[0121] The door (101) can be opened or closed by the user's physical force.

[0122] The door (101) can be opened or closed by electrical force provided by the electronic device (100).

[0123] The electronic device (100) may further include a door sensor.

[0124] At least one processor (120) can acquire (or receive) sensing data via a door sensor. At least one processor (120) can determine whether the door (101) is open based on the sensing data. At least one processor (120) can receive a preset signal depending on whether the door (101) is open.

[0125] When a signal indicating that the door (101) is open is received from the door sensor, at least one processor (120) can acquire a photographed image through the camera (190).

[0126] At least one processor (120) can obtain a shelf position based on the captured image.

[0127] A shelf location may indicate the physical placement of a shelf included in an electronic device (100). The shelf location may indicate where the shelf is placed in the electronic device (100). The shelf location may be described by the location of the shelf, identification information of the shelf, etc. The identification information of the shelf may include the shelf number, the name of the shelf, etc.

[0128] At least one processor (120) can identify a shelf object in the captured image and identify (or specify) a shelf location based on size information of the identified shelf object.

[0129] A shelf object may represent an image area representing a shelf. At least one processor (120) may identify a shelf object in a captured image based on preset features representing the shelf.

[0130] The electronic device (100) may include a plurality of shelves (10). The plurality of shelves (10) may be arranged vertically. Each of the plurality of shelves (10) may be a shelf for storing wine. The shelf may represent a wine rack, a wine holder, or a wine stand. A description of the structure of the plurality of shelves is described in FIG. 9.

[0131] At least one processor (120) can determine which shelf among the plurality of shelves holds the wine. At least one processor (120) can calculate the shelf location by specifying the shelf on which the wine is stored. The shelf location can indicate location information indicating which shelf among the plurality of shelves holds the wine. The operation of calculating the shelf location is described in FIG. 7.

[0132] At least one processor (120) can identify a shelf object in a captured image, obtain a horizontal length of the shelf object, and obtain a shelf position based on the horizontal length of the shelf object and a first mapping table stored in the memory (110). The first mapping table can include a horizontal length range corresponding to each of a plurality of shelves (10).

[0133] The shelf object may be rectangular in shape. A rectangular shape may include a rectangle, a square, a rhombus, a trapezoid, etc. In various embodiments, the shelf object may partially include a curved shape. The shelf object may be semicircular in shape.

[0134] The width of a shelf object can represent the length in the horizontal direction in a rectangular shape. The width of a shelf object can represent the length along a preset first axis or a preset first direction.

[0135] At least one processor (120) may obtain shelf positions from a first mapping table based on the horizontal length of the shelf object. The first mapping table may be a table indicating shelf positions corresponding to the horizontal length of the shelf object. Each of the plurality of shelves included in the electronic device (100) may have different placement heights. However, the placement positions of the cameras (190) that photograph the plurality of shelves may be fixed.

[0136] When the camera (190) captures at least one shelf among a plurality of shelves, the size displayed (or occupied) in the captured image may be different for each shelf.

[0137] For example, assume that the camera (190) is positioned at the topmost portion of the electronic device (100). The shelf positioned at the highest level (or position) may be displayed as a first size in the captured image. The shelf positioned at the lowest level (or position) may be displayed as a second size in the captured image. The farther the shelf is from the camera (190), the smaller it may appear in the captured image. The first size may be larger than the second size.

[0138] At least one processor (120) can identify that the shelf position is located further away from the camera (190) as the horizontal length is smaller.

[0139] At least one processor (120) can obtain first coordinate information corresponding to a shelf object based on a photographed image, and can obtain the length of a downward outline of the shelf object as the horizontal length of the shelf object based on the first coordinate information.

[0140] The first coordinate information may be information indicating the coordinates at which the shelf object is identified in the captured image. The coordinate information may be described as coordinates, coordinate data, location, location information, location data, etc. Since the shelf object is composed of multiple points, the first coordinate information indicating the shelf object may include multiple coordinate values.

[0141] According to one embodiment, the first coordinate information may include information about an area representing a shelf object. Since the shelf object is rectangular, the first coordinate information representing the shelf object may include coordinate values ​​representing both the outer edge and the interior of the rectangle. The first coordinate information may include coordinate values ​​for an area representing the shelf object.

[0142] According to one embodiment, the first coordinate information may include information about an outline representing a shelf object. Since the shelf object is rectangular, the first coordinate information representing the shelf object may include coordinate values ​​representing the outline of the rectangle. The first coordinate information may include coordinate values ​​for the outline representing the shelf object. The outline may be described as an edge, a border, a frame, or the like.

[0143] At least one processor (120) can identify an outline existing in a downward direction among a plurality of outlines representing a shelf object based on the first coordinate information. Since the shelf object has a rectangular shape, at least one processor (120) can identify four outlines representing the shelf object. At least one processor (120) can identify an outline existing in the lowest direction among the four outlines. The downward direction may refer to a downward direction on a captured image. When the lower left corner of the captured image is assumed to be the origin, the downward direction may refer to a direction close to the x-axis. The downward direction may refer to a direction close to the body of the electronic device (100).

[0144] The downward direction may be described as a preset direction, a preset axis, a direction inside the electronic device (100), a direction close to the body of the electronic device (100), etc. The operation of identifying the horizontal length of the outline corresponding to the downward direction is described in Fig. 10.

[0145] At least one processor (120) can obtain (or calculate) the horizontal length of the identified outline (or the specified outline). The at least one processor (120) can obtain the horizontal length of the outline and use it to identify a shelf location. The at least one processor (120) can calculate the horizontal length of the shelf object using various methods.

[0146] According to one embodiment, at least one processor (120) may calculate a horizontal length using a plurality of vertices representing a shelf object. At least one processor (120) may identify a plurality of vertices corresponding to the shelf object based on first coordinate information, identify two vertices adjacent to the camera (190) (or the body of the electronic device (100)) among the plurality of vertices, and obtain the length of an outline connecting the two vertices as the horizontal length of the shelf object. A description related thereto is described in FIG. 10.

[0147] According to one embodiment, at least one processor (120) may identify an outline having the smallest average y-axis value in the captured image among a plurality of outlines representing a shelf object. At least one processor (120) may obtain an average y-axis value for each outline by averaging coordinate information for the plurality of outlines. At least one processor (120) may identify an outline having the smallest average y-axis value among the plurality of average y-axis values.

[0148] According to one embodiment, at least one processor (120) can identify an outline of a plurality of outlines representing a shelf object that is closest to the body of the electronic device (100).

[0149] At least one processor (120) can identify (or specify or obtain) a shelf location based on the horizontal length of the shelf object and the first mapping table.

[0150] At least one processor (120) can identify a horizontal length range corresponding to a horizontal length among a plurality of horizontal length ranges included in the first mapping table, and identify a shelf corresponding to the identified horizontal length range as a shelf where wine is stored.

[0151] The first mapping table may be a table obtained by considering that shelf objects are displayed smaller when a shelf far from the camera (190) is captured. The first mapping table may include a horizontal length range in which each of a plurality of shelves is displayed in the captured image. At least one processor (120) may identify which of the horizontal length ranges included in the first mapping table includes the horizontal length of the shelf object captured in the captured image. At least one processor (120) may identify the shelf position based on the shelf corresponding to the identified horizontal length range. The first mapping table may vary depending on the type of the electronic device (100). This is because the heights of the plurality of shelves, the height of the camera, etc. vary depending on the type of device.

[0152] The first mapping table may be described as a first table, a length matching table, a shelf size mapping table, an image and shelf comparison table, a table for shelf specification, etc. A description of the first mapping table is provided in Fig. 13.

[0153] At least one processor (120) can obtain the location of the wine based on the captured image.

[0154] A wine location can indicate which area of ​​a shelf a wine is placed in. The wine location can be described as wine location information, wine location data, etc. The wine location can indicate the area where the wine is placed among multiple areas corresponding to the identified shelf. A description of the multiple areas corresponding to the shelf is provided in Fig. 14.

[0155] At least one processor (120) can identify a wine object in the captured image and obtain relative position information indicating the position of the wine object in the entire area of ​​the shelf based on the captured image.

[0156] A wine object may be an object representing wine. The wine object may be an object representing a wine bottle, a wine container, a wine storage container, etc. At least one processor (120) may determine whether a wine object is identified in a captured image acquired through a camera (190). To determine whether a wine object is included in the captured image, at least one processor (120) may consider preset features representing the wine object.

[0157] Once a wine object is identified, at least one processor (120) can identify an area within the entire shelf where the wine object is located based on the captured image. There may be various methods for identifying the area where the wine object is located.

[0158] According to one embodiment, at least one processor (120) may analyze the captured image to identify an area where a wine object is located. The electronic device (100) may pre-store a plurality of areas divided by coordinate values ​​for each of a plurality of shelves. Coordinate information for a plurality of areas corresponding to a specific shelf may be pre-stored in a memory (110), etc. At least one processor (120) may obtain a representative median corresponding to the wine object. At least one processor (120) may identify an area including the representative median among the coordinate information for the plurality of areas pre-stored in the memory (110). At least one processor (120) may obtain a wine location based on the identified area.

[0159] According to one embodiment, at least one processor (120) may obtain a wine position based on relative position information of the wine object. The relative position information may indicate where the wine object is relatively located on the shelf. The relative position information may include a value indicating the relative position of the wine object.

[0160] At least one processor (120) can obtain second coordinate information corresponding to a wine object based on the captured image. The second coordinate information can include a plurality of coordinate values ​​representing the wine object.

[0161] The second coordinate information may be information indicating the coordinates at which the wine object is identified in the captured image. The coordinate information may be described as coordinates, coordinate data, location, location information, location data, etc. Since a wine object is composed of multiple points, the second coordinate information indicating the wine object may include multiple coordinate values.

[0162] In one embodiment, the second coordinate information may include information about an area representing a wine object. Since the wine object is cylindrical, the second coordinate information representing the wine object may include coordinate values ​​representing both the outer edge and the interior of the cylinder. The second coordinate information may include coordinate values ​​for an area representing the wine object.

[0163] In one embodiment, the second coordinate information may include information about an outline representing a wine object. Since the wine object is cylindrical, the second coordinate information representing the wine object may include coordinate values ​​representing the outline of the cylinder. The second coordinate information may include coordinate values ​​for the outline representing the wine object. The outline may be described as an edge, a border, a border, or the like.

[0164] At least one processor (120) can obtain first coordinate information corresponding to a shelf object based on a photographed image. At least one processor (120) can obtain first coordinate information indicating an area including the shelf object in the entire area of ​​the photographed image.

[0165] At least one processor (120) can obtain the length of the downward outer line of the shelf object as the horizontal length of the shelf object based on the first coordinate information.

[0166] At least one processor (120) can obtain the minimum x-axis value of the shelf object based on the first coordinate information. At least one processor (120) can obtain the minimum value among x-axis values ​​representing the shelf object in the entire area of ​​the photographed image based on the first coordinate information.

[0167] At least one processor (120) can obtain second coordinate information corresponding to a wine object based on the photographed image. At least one processor (120) can obtain second coordinate information indicating an area including the wine object in the entire area of ​​the photographed image.

[0168] At least one processor (120) can obtain at least one of the minimum x-axis value of the wine object or the maximum x-axis value of the wine object based on the second coordinate information. At least one processor (120) can obtain the minimum value among the x-axis values ​​representing the wine object based on the second coordinate information.

[0169] At least one processor (120) can obtain relative position information corresponding to the wine object based on at least one of the minimum x-axis value of the wine object or the maximum x-axis value of the wine object, the minimum x-axis value of the shelf object, and the horizontal length of the shelf object. At least one processor (120) can obtain the wine position based on the relative position information.

[0170] According to various embodiments, referring to FIG. 17, a region representing a shelf object may be separated to calculate relative position information. At least one processor (120) may separate and calculate only the region representing the shelf object to calculate a value related to the x-axis. At least one processor (120) may calculate values ​​related to the x-axis based on first coordinate information representing the region related to the shelf object. When calculating relative position information by separating the region representing the shelf object, at least one processor (120) may calculate the minimum x-axis value of the shelf object as 0, and may calculate the minimum x-axis value of the wine object or the maximum x-axis value of the wine object based on the region represented by the first coordinate information.

[0171] The minimum x-axis value of the wine object, the maximum x-axis value of the wine object, the minimum x-axis value of the shelf object, the horizontal length of the shelf object, etc. are described in Fig. 15. The process of calculating relative position information is described in Fig. 16.

[0172] At least one processor (120) can obtain the wine position based on the relative position information and a second mapping table stored in the memory (110). The second mapping table can include a relative position range corresponding to each of a plurality of areas of the shelf.

[0173] The second mapping table may include relative position ranges corresponding to each of a plurality of areas corresponding to the shelf.

[0174] The second mapping table may vary depending on the type of shelf. If the shelves have the same structure, the same second mapping table can be used. However, shelves with different structures may use different second mapping tables. A description of the second mapping table is provided in Fig. 16.

[0175] The structure of the shelf may represent the structure of a single shelf itself. For example, according to the structure of the shelf described in the embodiment (1410) of FIG. 14, there may be seven areas (or storage areas).

[0176] At least one processor (120) can obtain wine characteristic information corresponding to the wine. At least one processor (120) can identify a wine object based on a photographed image and obtain wine characteristic information corresponding to the identified wine object.

[0177] Wine characteristic information may include information describing the characteristics of the wine. Wine characteristic information may include descriptive information about the wine being stored. Wine characteristic information may include various information related to the wine.

[0178] Wine characteristic information may include at least one of the following: name, type, flavor, manufacturer, date of production, expiration date, price, or rating. However, this is not limited to this, and wine characteristic information may include various additional information describing the wine.

[0179] At least one processor (120) can obtain wine characteristic information based on various routes.

[0180] According to one embodiment, at least one processor (120) may acquire wine characteristic information from the electronic device (100) itself. The electronic device (100) may store wine characteristic information corresponding to a plurality of wines in a memory (110), etc. At least one processor (120) may acquire (or identify) wine characteristic information corresponding to a wine object identified based on a photographed image among the plurality of wine characteristic information stored in the memory (110).

[0181] According to one embodiment, at least one processor (120) may obtain wine characteristic information from a server (200). The server (200) may include a database storing wine characteristic information corresponding to each of a plurality of wines. The server (200) may be described as an external server.

[0182] At least one processor (120) may be connected to a server (200) via a communication interface (130) included in the electronic device (100). At least one processor (120) may transmit information to or receive information from the server (200) via the communication interface (130).

[0183] At least one processor (120) can utilize various input data to obtain wine characteristic information.

[0184] According to one embodiment, at least one processor (120) may acquire wine characteristic information using an original photographed image. At least one processor (120) may identify a wine object based on the photographed image and acquire wine characteristic information based on an area corresponding to the identified wine object.

[0185] According to one embodiment, at least one processor (120) may identify a wine object based on a captured image and obtain a wine image by isolating (or extracting or cropping) only the area containing the wine object. The wine image may represent separate image data containing only the wine object. At least one processor (120) may obtain wine characteristic information based on the wine image.

[0186] According to one embodiment, at least one processor (120) may acquire wine characteristic information using pre-stored wine sample images. At least one processor (120) may store wine sample images in a database through a wine registration process, and when a wine object is identified in the captured image, acquire a wine sample image corresponding to the wine object, and acquire wine characteristic information based on the wine sample image.

[0187] The wine registration process may include a process of acquiring a photographic image including a wine object larger than a threshold size using a camera (190) included in an electronic device (100). Since the position of the camera (190) is fixed, the resolution of the photographed image acquired by at least one processor (120) through the camera (190) may be the same. However, the size of the wine object included in the photographed image may vary depending on the location where the wine is captured.

[0188] For example, a captured image of a wine object at a close distance from the camera (190) may include wine objects larger than a threshold size. However, a captured image of a wine object at a far distance from the camera (190) may include wine objects smaller than a threshold size.

[0189] During the wine registration process, at least one processor (120) can identify the size of the wine object in the acquired photographed image. At least one processor (120) can identify whether the size of the wine object is greater than a threshold value. At least one processor (120) can obtain coordinate information of the wine object in the photographed image and determine the size of the wine object based on the coordinate information.

[0190] The size of the wine object may include at least one of the vertical length, horizontal length, diagonal length, and width of the wine object. If the size of the wine object is greater than or equal to a threshold value, at least one processor (120) may store the captured image as a wine sample image in a database. The database may refer to a separate storage space (or area) in the memory (110) that stores only wine sample images.

[0191] If the size of the wine object is below a threshold value, at least one processor (120) may provide a guide UI (User Interface) requesting re-capture. The guide UI may include at least one of information indicating that the size of the wine object is small, information guiding the user to take a picture of the wine closer to the camera (190), and information guiding the user to input a re-capture command.

[0192] Through the guide UI, the user can take a picture for registration while moving the wine closer to the camera (190). Through the guide UI, the user can easily save a wine sample image containing a wine object larger than a threshold size.

[0193] According to various embodiments, the wine registration process may include registering a photographed image captured by a camera included in a terminal device (300). A user may capture a photograph of wine within a threshold distance using the terminal device (300), and store the image captured within the threshold distance as a wine sample image in a database.

[0194] The camera (190) of the electronic device (100) can acquire a first camera that acquires a photographed image based on a first resolution, and a second camera that acquires a photographed image based on a second resolution smaller than the first resolution.

[0195] During the wine registration process, at least one processor (120) can acquire a wine sample image using the first camera.

[0196] During the wine storage process (normal process), at least one processor (120) can acquire a photographed image using a second camera.

[0197] The wine sample image acquired through the first camera may include a wine object of a larger size than the image captured using the second camera. Descriptions of the wine registration process are provided in FIGS. 25 to 27.

[0198] At least one processor (120) may determine whether wine characteristic information is obtained based on various input data. If the size of a wine object included in the input data (e.g., a wine image) is too small or the wine object is not accurately identified, the at least one processor (120) may not be able to obtain wine characteristic information. Furthermore, if the wine is a new wine that is not stored in a previously stored database corresponding to the wine, the at least one processor (120) may not be able to obtain wine characteristic information.

[0199] At least one processor (120) may provide a guide UI for obtaining wine characteristic information when an event occurs in which wine characteristic information cannot be obtained. The guide UI may include guide information for entering wine characteristic information.

[0200] A guide UI may be provided on an electronic device (100). A description thereof is provided in FIG. 31.

[0201] A guide UI may be provided on a terminal device (300). A description related thereto is provided in Fig. 32.

[0202] A description of the guide UI is provided in Figure 33.

[0203] At least one processor (120) may obtain wine storage information including shelf locations and wine locations, and may obtain wine characteristic information corresponding to the stored wine. At least one processor (120) may obtain wine information including wine storage information and wine characteristic information, and may store the wine information in the memory (110).

[0204] At least one processor (120) can provide stored wine information to a user. Various methods may be used to provide the information to the user.

[0205] According to one embodiment, the electronic device (100) can provide wine information to a user by executing an application included in the electronic device (100). The application can generate and provide a screen indicating the wine storage status.

[0206] The electronic device (100) may include a display (140). The electronic device (100) may display wine information on the display (140). The electronic device (100) may display a screen provided by a wine application including wine information on the display (140).

[0207] According to one embodiment, the electronic device (100) can transmit stored wine information to the user's terminal device (300). The terminal device (300) can receive the wine information from the electronic device (100). The wine information can be provided to the user through a wine application installed in the terminal device (300). A screen related to the wine application providing wine information can be displayed through a display included in the terminal device (300).

[0208] According to one embodiment, the electronic device (100) may transmit stored wine information to an external server. The external server may be an Internet of Things (IoT) server. The external server may be connected to at least one home appliance (or smart home appliance). The external server may transmit various control commands to at least one home appliance connected to the external server. The external server may receive various information from at least one home appliance. The external server may be connected to a terminal device (300). The external server may provide wine information in response to a request from the terminal device (300). A description thereof is provided in FIG. 30.

[0209] The electronic device (100) can provide information about the location of the wine shelf and the wine's position on the shelf when storing wine. The storage location and information about the wine can be automatically stored together without the user having to run a specific application to directly enter information about the wine. The user can easily receive information about wine storage and characteristics without any separate action.

[0210] The electronic device (100) can provide various services using wine characteristic information.

[0211] According to one embodiment, after acquiring wine characteristic information, the electronic device (100) may provide an optimal temperature corresponding to the wine based on the wine characteristic information. The optimal temperature corresponding to the wine characteristic information may vary for each wine. The electronic device (100) may identify an optimal temperature corresponding to the wine based on the wine characteristic information and perform a refrigeration function by setting the identified optimal temperature.

[0212] According to one embodiment, after acquiring wine characteristic information, the electronic device (100) may provide recommended food information corresponding to the wine characteristic information. The electronic device (100) may identify recommended food corresponding to the wine characteristic information and provide the recommended food to the user. The time of acquiring the wine characteristic information may be the time of storing the wine. The electronic device (100) may identify whether an event has occurred in which the user takes out the stored wine. If an event in which the user takes out the stored wine has occurred, the electronic device (100) may acquire wine characteristic information of the wine corresponding to the occurrence of the event. The electronic device (100) may identify recommended food information corresponding to the wine characteristic information and provide the recommended food information to the user. The recommended food information corresponding to the wine characteristic information may be stored in advance.

[0213] In one embodiment, after acquiring wine characteristic information, the electronic device (100) can compare wines stored on all shelves and guide (or recommend) wine storage locations. Storage temperatures may vary for each wine. Therefore, storing wines with similar storage temperatures on the same shelf may be necessary for wine management. The electronic device (100) can identify the optimal temperature for each wine based on the wine characteristic information.

[0214] The electronic device (100) can compare the optimal temperature of the already stored wine with the optimal temperature of the newly stored wine. If the difference between the optimal temperature of the already stored wine and the optimal temperature of the newly stored wine is less than a threshold value, the electronic device (100) can guide the newly stored wine to be moved to the shelf where the already stored wine is located.

[0215] Assume that the wine already stored is described as the first wine, and that the first wine is stored in the first area of ​​the first shelf. An event occurs in which the second wine is newly stored, and that the difference between the optimal temperature of the first wine and the optimal temperature of the second wine is less than a threshold value. After obtaining wine characteristic information of the second wine, the electronic device (100) can provide information (or a screen) guiding the storage of the second wine in the second area of ​​the first shelf where the first wine is stored.

[0216] The electronic device (100) can update the wine storage location. When the door is opened, the electronic device (100) can identify whether the location of the stored wine has changed. If an event indicating a change in the location of the stored wine is identified, the electronic device (100) can update existing stored information (storage location information for all stored wines) using the acquired wine storage information. The electronic device (100) can update only the location (or area) where the wine location has changed.

[0217] The electronic device (100) can identify an event in which wine moves from its storage location. The electronic device (100) can track the location of the wine in real time through the camera (190). The electronic device (100) can identify whether the wine is continuously identified in multiple captured images acquired through the camera (190).

[0218] When an event in which wine is continuously identified and moves from a first location to a second location is identified, the electronic device (100) can utilize the wine characteristic information corresponding to the wine stored in the first location as is. Since the wine has only moved from the first location to the second location, the electronic device (100) can utilize the wine characteristic information stored together with the wine storage information including the first location as is. The electronic device (100) can store the wine characteristic information together with the wine storage information including the second location.

[0219] If an event is identified in which the wine is discontinuously identified and moves from a first location to a second location, the electronic device (100) may provide a notification to the user. Discontinuous identification of the wine may indicate that the wine object is not identified in some of the continuously acquired captured images. For example, a user may start by taking out a wine within the field of view of the camera (190), move the wine out of the field of view, and then move the wine back into the field of view after a preset period of time. The user may take the wine from the first location, move the wine out of the field of view of the camera (190), and then place the wine back in the second location.

[0220] The electronic device (100) can obtain the time during which the wine object is not captured in the captured image. The time during which the wine object is not captured in the captured image can be described as the wine non-detection time.

[0221] If the wine non-detection time is less than or equal to the first threshold time, the electronic device (100) may provide first notification information. The first notification information may include voice feedback. If the wine is new, the first notification information may include a guide command to touch the wine once.

[0222] If the wine non-detection time exceeds the first threshold time and is less than or equal to the second threshold time, the electronic device (100) may provide a guide UI to the terminal device (300). The guide UI may include information querying whether the newly recognized wine at the second location is a wine previously taken from the first location or a new wine. The second threshold time may be greater than the first threshold time.

[0223] Based on user input received through the guide UI, the electronic device (100) can determine whether the wine stored in the second location is an existing wine or a new wine. If the wine stored in the second location is an existing wine, the electronic device (100) can directly utilize the wine characteristic information of the wine stored in the first location. If the wine stored in the second location is a new wine, the electronic device (100) can newly acquire wine characteristic information based on the wine image of the wine stored in the second location.

[0224] If the wine non-detection time exceeds the second threshold time, the electronic device (100) can determine that the wine stored in the second location is new wine rather than existing wine. The electronic device (100) can obtain new wine characteristic information based on the wine image of the wine stored in the second location.

[0225] FIG. 3 is a block diagram for explaining the configuration of the electronic device (100) of FIG. 2, according to one embodiment.

[0226] Referring to FIG. 3, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).

[0227] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). The memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0228] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0229] The memory (110) can store at least one instruction. Based on the instruction stored in the memory (110), at least one processor (120) can perform various operations.

[0230] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (120) may perform various functions by executing computer executable instructions stored in a memory.

[0231] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0232] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0233] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.

[0234] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0235] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0236] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0237] According to various embodiments, the communication interface (130) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

[0238] According to various embodiments, the communication interface (130) may utilize different communication modules to communicate with external devices, such as remote control devices, and external servers. For example, the communication interface (130) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is merely an embodiment, and the communication interface (130) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.

[0239] The display (140) may be implemented as a variety of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. According to an embodiment of the present disclosure, the display (140) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor for detecting user interaction.

[0240] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0241] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). An output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0242] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0243] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0244] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0245] The camera (190) is a device configured to capture a subject and generate a captured image, and the captured image includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, lens, infrared sensor, or the like.

[0246] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0247] FIG. 4 is a diagram for explaining an operation of storing wine information according to one embodiment.

[0248] Referring to FIG. 4, the electronic device (100) may include at least one of a door sensor (410), a door opening recognition module (420), an image sensor (430), a storage location detection model (440), a wine characteristic information retrieval model (450), or a wine information storage module (460).

[0249] The door sensor (410) can sense whether the door attached to the electronic device (100) is open or closed.

[0250] According to one embodiment, the door sensor (410) may generate a first signal when the door is open. The electronic device (100) may identify the door as open only when the first signal is received. If the first signal is not received, the electronic device (100) may identify the door as closed.

[0251] The door sensor (410) can obtain sensing data. The electronic device (100) can identify whether the door is open or closed based on the sensing data.

[0252] According to one embodiment, the door sensor (410) may generate a first signal when the door is open and a second signal when the door is closed. When the first signal is received, the electronic device (100) may identify that the door is open. When the second signal is received, the electronic device (100) may identify that the door is closed.

[0253] When the door is opened, the door sensor (410) can transmit a door open notification (or first signal) indicating that the door is open to the door open recognition module (420). When the door open notification is received, the door open recognition module (420) can transmit a shooting command to the image sensor (430).

[0254] The image sensor (430) may include a camera (190). When a shooting command is received, the image sensor (430) may acquire sensing data. The sensing data may include a captured image. The captured image may be an image captured of the front of the electronic device (100). The image sensor (430) may be positioned toward the front side of the electronic device (100) (the side from which the door can be opened). A description related to the captured image is described in FIG. 1.

[0255] The captured image acquired through the image sensor (430) can be transmitted to the storage location detection model (440). The electronic device (100) can transmit the captured image to the storage location detection model (440).

[0256] The storage location detection model (440) may include at least one of a shelf location detection model (441) or a wine location detection model (442). The storage location detection model (440) may include information related to the storage of wine.

[0257] The shelf location detection model (441) can obtain shelf locations (or shelf location information) indicating which shelf among multiple shelves the wine is stored on. The shelf location detection model (441) can transmit the shelf locations to the wine information storage module (460). Descriptions related to shelf locations are described in FIGS. 9 to 13.

[0258] The wine position detection model (442) can obtain wine positions (or wine position information) indicating where wine is placed on a specific shelf. The wine position detection model (442) can transmit the wine positions to the wine information storage module (460). Descriptions related to wine positions are described in FIGS. 14 to 18 .

[0259] The electronic device (100) can obtain the shelf position through the shelf position detection model (441) and the wine position through the wine position detection model (442).

[0260] The electronic device (100) can transmit a photographed image or wine object area to a wine feature information retrieval model (450).

[0261] According to one embodiment, the electronic device (100) may transmit a photographed image including a wine object to a wine feature information retrieval model (450). The photographed image may include a shelf object and a wine object.

[0262] According to one embodiment, the electronic device (100) may extract (or crop) an image region containing only a wine object from a captured image. The electronic device (100) may transmit data regarding the image region containing only a wine object to a wine feature information retrieval model (450). The data regarding the image region containing only a wine object may be described as a wine image.

[0263] There may be various ways for the wine feature information retrieval model (450) to receive a photographed image or a wine image.

[0264] According to one embodiment, the storage location detection model (440) can transmit the received photographed image (or wine image) to the wine feature information retrieval model (450).

[0265] According to one embodiment, the wine location detection model (442) can transmit the received photographed image (or wine image) to the wine feature information retrieval model (450). The wine location detection model (442) can analyze the location of the wine in the photographed image. The wine location detection model (442) can obtain the wine image to analyze the location of the wine and transmit the wine image to the wine feature information retrieval model (450).

[0266] According to one embodiment, the image sensor (430) can transmit the captured image to the wine feature information retrieval model (450).

[0267] The wine characteristic information retrieval model (450) can receive a photographed image or a wine image. The wine characteristic information retrieval model (450) can analyze the characteristics of the wine based on the photographed image or the wine image. The wine characteristic information retrieval model (450) can obtain wine characteristic information. The wine characteristic information can include various information about the wine itself. The wine characteristic information retrieval model (450) can transmit the wine characteristic information to the wine information storage module (460). Descriptions related to the wine characteristic information are described in FIGS. 19 and 20 .

[0268] The wine information storage module (460) may be a module that stores various information related to wine. The wine information storage module (460) may be a module corresponding to a wine application. The wine information storage module (460) may store wine information. The wine information may include at least one of wine storage information indicating where the wine is stored and wine characteristic information related to the characteristics of the wine. The wine storage information may include at least one of shelf location and wine location.

[0269] The wine information storage module (460) can receive shelf locations from the shelf location detection model (441). The wine information storage module (460) can receive wine locations from the wine location detection model (442). The wine information storage module (460) can receive wine characteristic information from the wine characteristic information retrieval model (450).

[0270] The wine information storage module (460) can provide wine information (at least one of wine storage information or wine characteristic information) according to a preset control command.

[0271] FIG. 5 is a diagram for explaining an operation of obtaining wine storage information and wine characteristic information according to one embodiment.

[0272] Referring to FIG. 5, the electronic device (100) can identify whether the door is open (S510). If the door is not open (S510-N), the electronic device (100) can repeat step S510 until the door is opened.

[0273] When the door is opened (S510-Y), the electronic device (100) can obtain wine storage information (S515). The wine storage information may include information related to wine storage. The wine storage information may include at least one of a shelf location indicating information related to the shelf on which the wine is placed, or a wine location indicating which area of ​​a specific shelf the wine is placed in.

[0274] The electronic device (100) can acquire wine characteristic information (S560). The wine characteristic information can represent various information related to the characteristics of the wine. The wine characteristic information can include at least one of the following: name, type, flavor, manufacturer, manufacturing date, expiration date, price, and rating.

[0275] Wine characteristics can vary from wine to wine. Even wines with the same name from the same manufacturer may have different production dates and expiration dates.

[0276] The electronic device (100) can store wine information including at least one of wine storage information or wine characteristic information (S570).

[0277] FIG. 6 is a diagram for explaining an operation of obtaining wine storage information according to one embodiment.

[0278] Steps S610, S615, S660, and S670 of FIG. 6 may correspond to steps S510, S515, S560, and S570 of FIG. 5.

[0279] When the door opening is identified (S610-Y), the electronic device (100) can acquire a photographed image (S620). The electronic device (100) can acquire a shelf position from the photographed image (S630). The shelf position can be determined based on the position of the shelf object. The electronic device (100) can acquire coordinate information of the shelf object and, based on the coordinate information of the shelf object, determine which shelf among the plurality of shelves the shelf object is.

[0280] For example, the electronic device (100) can acquire the shelf position based on the horizontal length of the shelf object included in the captured image. Assume that the shelf is divided into multiple layers. A shelf placed on a relatively higher layer may appear larger in the captured image. A shelf placed on a relatively lower layer may appear smaller in the captured image. A description related to this is provided in FIG. 11.

[0281] The electronic device (100) can obtain the location of the wine based on the captured image (S640). The electronic device (100) can identify which area of ​​the entire area of ​​a specific shelf the wine is placed in. The wine location can indicate the area (or location) where the wine is placed on a specific shelf.

[0282] The electronic device (100) can store wine storage information including at least one of a shelf location or a wine location (S650).

[0283] FIG. 7 is a drawing for explaining an operation of obtaining a shelf position according to one embodiment.

[0284] Referring to FIG. 7, steps S710, S715, S760, and S770 may correspond to steps S510, S515, S560, and S570 of FIG. 5. Steps S710, S715, S720, S730, S740, S750, S760, and S770 may correspond to steps S610, S615, S620, S630, S640, S650, S660, and S670 of FIG. 6.

[0285] The electronic device (100) can identify a shelf object in a photographed image (S731). The electronic device (100) can obtain the horizontal length of the shelf object (S732).

[0286] The electronic device (100) can obtain a shelf position based on the horizontal length of the shelf object and the first mapping table (S733). The first mapping table may be a table indicating shelf positions corresponding to horizontal lengths. The first mapping table may be a mapping table required to specify a shelf position based on the horizontal length of the shelf object. The first mapping table may include a length range corresponding to each of a plurality of shelf positions. If the horizontal length of the shelf object falls within a specific length range, the electronic device (100) can identify the shelf as corresponding to the specific length range. A description related to the first mapping table is described in FIG. 13.

[0287] According to one embodiment, the electronic device (100) can identify a shelf object on which wine is placed. The electronic device (100) can determine whether both the shelf object and the wine object are recognized in the captured image. If both the shelf object and the wine object are identified, the electronic device (100) can identify the shelf object on which the wine object is placed.

[0288] In one embodiment, when a wine object is identified, the electronic device (100) can determine whether a shelf object on which the wine object is placed is identified.

[0289] According to one embodiment, the electronic device (100) can determine whether to acquire a shelf position by comparing the horizontal and vertical lengths of the shelf object. The electronic device (100) can compare the horizontal and vertical lengths of the shelf object in real time by acquiring a captured image in real time. The electronic device (100) can identify a bounding box based on coordinate information representing the shelf object. The electronic device (100) can acquire the horizontal and vertical lengths of the bounding box representing the shelf object.

[0290] If the value (or ratio) of the vertical length divided by the horizontal length is less than or equal to a threshold value (e.g., 0.5), the electronic device (100) may not perform a calculation operation to obtain a shelf position. If the value (or ratio) of the vertical length divided by the horizontal length exceeds a threshold value (e.g., 0.5), the electronic device (100) may perform a calculation operation to obtain a shelf position.

[0291] FIG. 8 is a diagram for explaining an operation of obtaining a wine position according to one embodiment.

[0292] Referring to FIG. 8, steps S810, S815, S820, S830, S840, S850, S860, and S870 may correspond to steps S510, S515, S560, and S570 of FIG. 5. Steps S810, S815, S820, S830, S840, S850, S860, and S870 may correspond to steps S610, S615, S620, S630, S640, S650, S660, and S670 of FIG. 6. Steps S810, S815, S820, S830, S840, S850, S860, S870 may correspond to steps S710, S715, S720, S730, S740, S750, S760, S770 of FIG. 7.

[0293] Referring to FIG. 8, the electronic device (100) can identify a wine object in a photographed image (S841). The electronic device (100) can obtain at least one of the minimum x-axis value (x1) of the wine object, the maximum x-axis value (x2) of the wine object, and the horizontal length of the shelf object (S842).

[0294] The x-axis of the wine object may be described as the first axis. The electronic device (100) may obtain coordinate information representing the wine object. The electronic device (100) may obtain the minimum value (x1) of the x-axis among the coordinates representing the wine object. The electronic device (100) may obtain the maximum value (x2) of the x-axis among the coordinates representing the wine object. A description related thereto is described in FIG. 15.

[0295] The electronic device (100) can obtain relative position information of the wine based on at least one of the minimum x-axis value (x1) of the wine object, the maximum x-axis value (x2) of the wine object, and the horizontal length of the shelf object (S843).

[0296] Relative position information can indicate the relative position of a wine object within the entire shelf area. Relative position information can be determined within the range of 0 to 1. For example, if a wine is placed on the far left of a shelf, the relative position information can be 0. If a wine is placed on the far right of a shelf, the relative position information can be 1.

[0297] The electronic device (100) can obtain the wine's location based on the wine's relative location information and a second mapping table (S844). The second mapping table may be a mapping table used to specify the wine's location. The second mapping table may include information used to specify a location corresponding to the relative location information. A description related to this is provided in FIG. 16.

[0298] FIG. 9 is a drawing for explaining the structure of a shelf included in an electronic device (100), according to one embodiment.

[0299] Referring to FIG. 9, the electronic device (100) may include a plurality of shelves (10-1, 10-2, 10-3, 10-4, 10-5, 10-6, 10-7, 10-8, 10-9, 10-10, 10-11, 10-12). Each shelf may be arranged vertically. The position of each shelf may have a different arrangement structure depending on the type of electronic device (100). The camera (190) may be arranged on the front of the electronic device (100). The camera (190) may be arranged in a direction for photographing the plurality of shelves (10-1 to 10-12). The camera (190) may be arranged to face the floor with respect to the horizontal direction.

[0300] A plurality of shelves (10-1 to 10-12) can be arranged inside the electronic device (100) when the door (30) of the electronic device (100) is closed. When the door (30) is open, the plurality of shelves (10-1 to 10-12) can be moved (or slid) from the inside to the outside of the electronic device (100). Conversely, the plurality of shelves (10-1 to 10-12) can be moved from the outside to the inside of the electronic device (100).

[0301] According to one embodiment, the plurality of shelves (10-1 to 10-12) can be moved by a physical force applied by a user.

[0302] According to one embodiment, the plurality of shelves (10-1 to 10-12) can be automatically moved by a motor included in the electronic device (100).

[0303] FIG. 10 is a drawing for explaining an operation of identifying a shelf object through an image, according to one embodiment.

[0304] According to the embodiment (1010) of FIG. 10, the electronic device (100) can identify a shelf object in a captured image. The electronic device (100) can identify coordinate information representing the shelf object. The electronic device (100) can obtain coordinate information (1010-B) of the shelf object. The coordinate information (1010-B) can include a bounding box.

[0305] The electronic device (100) can obtain vertex coordinates (1011, 1012, 1013, 1014) based on coordinate information representing the shelf object. The shelf object identified in the captured image can include four vertices. A first vertex (1011), a second vertex (1012), a third vertex (1013), and a fourth vertex (1014) can be identified.

[0306] The electronic device (100) can identify two vertices that are located relatively inside the electronic device (100) among a plurality of vertices. The two vertices that are located relatively inside may be a first vertex (1011) and a second vertex (1012).

[0307] The electronic device (100) can identify two vertices that are located relatively outside of the electronic device (100) among a plurality of vertices. The two vertices that are located relatively outside may be a third vertex (1013) and a fourth vertex (1014).

[0308] The electronic device (100) can obtain the horizontal length (d1) of the lower side of the shelf object connecting the first vertex (1011) and the second vertex (1012).

[0309] The electronic device (100) can obtain the upper side length (d2) of the shelf object connecting the third vertex (1013) and the fourth vertex (1014).

[0310] According to the embodiment (1020) of FIG. 10, the electronic device (100) can obtain coordinate information (1020-B) of a shelf object in a rectangular shape. The electronic device (100) can obtain coordinate information (1020-B) representing the shelf object based on the horizontal length (d1) of the lower side of the shelf object.

[0311] The electronic device (100) can obtain the horizontal length (d1) of the lower side of the shelf object connecting the first vertex (1021) and the second vertex (1022). The electronic device (100) can generate coordinate information (1020-B) of the shelf object so that the horizontal length (d1) of the lower side of the shelf object and the horizontal length of the upper side of the shelf object are the same. The electronic device (100) can determine the positions of the third vertex (1023) and the fourth vertex (1024) so ​​that the horizontal length (d1) of the lower side of the shelf object and the horizontal length of the upper side of the shelf object are the same.

[0312] FIG. 11 is a drawing for explaining an operation of identifying a shelf position through an image, according to one embodiment.

[0313] Referring to FIG. 11, the electronic device (100) can obtain coordinate information of a shelf object. The coordinate information may be in a rectangular shape, as in the embodiment (1020) of FIG. 10. The electronic device (100) can obtain the horizontal length of the shelf object based on the coordinate information. The electronic device (100) can specify a shelf position based on the horizontal length of the shelf object. Since the shelf position and the position of the camera (190) are fixed, the electronic device (100) can specify a shelf position (floor) based on the horizontal length of the shelf object.

[0314] Referring to the embodiment (1110) of FIG. 11, based on a photographed image of the 12th shelf (10-12 of FIG. 10), the electronic device (100) can obtain coordinate information (1110-B) of the shelf object. Based on the coordinate information (1110-B), the electronic device (100) can obtain the horizontal length (482 pixels) of the shelf object. Based on the horizontal length (482 pixels), the electronic device (100) can identify that the shelf object is the 12th shelf (10-12 of FIG. 10).

[0315] Referring to the embodiment (1120) of FIG. 11, based on a photographed image of the first shelf (10-1 of FIG. 10), the electronic device (100) can obtain coordinate information (1120-B) of the shelf object. Based on the coordinate information (1120-B), the electronic device (100) can obtain the horizontal length (119 pixels) of the shelf object. Based on the horizontal length (119 pixels), the electronic device (100) can identify that the shelf object is the first shelf (10-1 of FIG. 10).

[0316] Figure 12 is a drawing for explaining the point of identifying a shelf object.

[0317] Referring to FIG. 12, the electronic device (100) can identify the movement of a shelf object in real time. The electronic device (100) can determine in real time whether the shelf object is identified in the captured image. The electronic device (100) can acquire the captured image in real time from the time the door is opened. The electronic device (100) can determine whether the shelf object is identified in the captured image.

[0318] Since the shelf is fixed to the electronic device (100), the shelf object can always be the same. The electronic device (100) may store information about the shelf object in advance. The electronic device (100) can identify the existence of the shelf object based on only partial characteristics, even if the entire shape of the shelf object is not recognized.

[0319] For example, a preset feature object (1201) may exist on the upper edge of a shelf. The preset feature object (1201) may be an object representing the upper edge of the shelf object. The preset feature object (1201) may include a horizontal line object of a preset color and / or preset thickness.

[0320] When a preset feature object (1201) is identified as being included in the captured image, the electronic device (100) can determine that a shelf object has been identified. The electronic device (100) can identify in real time a situation in which a shelf exists inside the electronic device (100) and then moves (or slides) to the outside.

[0321] The embodiment (1210) of FIG. 12 may represent a captured image acquired at a second point in time, which is after the first point in time when the door is opened. The captured image acquired at the second point in time may include only a portion of the entire area of ​​the shelf object. However, the captured image may include a preset feature object (1201). The electronic device (100) may recognize the shelf object by identifying the preset feature object (1201) in the captured image.

[0322] Example 1220 of FIG. 12 may represent a captured image acquired at a third point in time, which is after the second point in time. The shelf may have moved further outward at the third point in time than at the second point in time. More shelf objects may be identified in the captured image acquired at the third point in time than in the captured image acquired at the second point in time.

[0323] Example 1230 of FIG. 12 may represent a captured image acquired at a fourth point in time, which is after the third point in time. The shelf may have been moved further outward at the fourth point in time than at the third point in time. Example 1230 may represent that the shelf has been moved all the way.

[0324] Example 1240 of FIG. 12 may represent a captured image acquired at a fifth point in time, which is after the fourth point in time. The fifth point in time may be a point in time after the user places the wine on the shelf. The captured image acquired at the fifth point in time may include a wine object (1202).

[0325] The timing of calculating shelf positions can vary. The timing of performing calculations for shelf positions can vary.

[0326] According to an embodiment, when a preset object (1201) is identified, the electronic device (100) can calculate a shelf position. The electronic device (100) can identify a portion of the shelf object according to the preset object (1201). The electronic device (100) can specify the shelf position based on the horizontal length of the lower side of the portion of the shelf object. Regardless of embodiments (1210 to 1240), the horizontal length of the lower side of the shelf object identified in the captured image can be constant. Even if not all portions of the shelf object are included in the captured image, as in embodiments (1210, 1220), the electronic device (100) can calculate the shelf position.

[0327] According to one embodiment, if the shelf object is stationary, the electronic device (100) can calculate the shelf position. If the movement of the shelf object stops, the electronic device (100) can calculate the shelf position. If all parts of the shelf object are included in the captured image, as in embodiment (1230), the electronic device (100) can calculate the shelf position.

[0328] In one embodiment, if both a shelf object and a wine object are included in the captured image, the electronic device (100) can identify the shelf object.

[0329] In one embodiment, when a shelf object is identified in the captured image, the electronic device (100) can identify whether a wine object is included in the captured image. When the wine object is identified, the electronic device (100) can calculate a shelf position.

[0330] In one embodiment, when a wine object is identified in the captured image, the electronic device (100) can identify whether a shelf object is included in the captured image. When a shelf object is identified, the electronic device (100) can calculate a shelf position.

[0331] The wine object can also be moved in the same manner as the shelf object described in Fig. 12. While the shelf object moves along a fixed path with a fixed size (assuming the same shelf), the wine object can move along a different path depending on the user. The electronic device (100) can determine whether the wine object is moving or stationary.

[0332] The electronic device (100) can determine whether the position of the wine object is fixed for a preset period of time (or a threshold period of time) based on a plurality of captured images acquired in real time. If the position of the wine object is fixed for a preset period of time, the electronic device (100) can determine that the wine object has been identified. If the position of the wine object is fixed for a preset period of time, the electronic device (100) can identify the position of the shelf object. If the wine object is fixed for a preset period of time and the shelf object is identified, the electronic device (100) can calculate the shelf position.

[0333] Figure 13 is a drawing for explaining a table including information for specifying shelf positions.

[0334] The first mapping table (1310) of FIG. 13 may represent a mapping table used to specify shelf positions. The first mapping table (1310) may include information on the horizontal length recognized in the photographed image according to the placement position (or placement height) of the shelves. d may represent the actually measured horizontal length of the shelf. d may represent the horizontal length of the shelf (e.g., the 12th shelf) located at the highest position in the photographed image. The unit of d may be mm, cm, pixel, etc.

[0335] The lower the shelf position, the smaller the shelf width measured in the photographed image. This is described in Fig. 11.

[0336] The range (lower limit, upper limit) corresponding to each shelf described in the first mapping table may vary depending on the type of shelf included in the electronic device (100), the type of camera, camera settings (e.g., resolution, magnification, etc.), etc. The constant value multiplied by d in the first mapping table may vary depending on the user's settings.

[0337] The electronic device (100) can obtain the horizontal length of the shelf object. The electronic device (100) can identify a shelf corresponding to the horizontal length of the shelf object by considering the first mapping table (1310). For example, if the horizontal length of the shelf object is identified as 0.18d, the electronic device (100) can identify the shelf object as the second shelf.

[0338] FIG. 14 is a diagram illustrating an operation of identifying a wine location through an image, according to one embodiment.

[0339] Referring to the embodiment (1410) of FIG. 14, the shelf may include multiple areas where wine can be placed. The shelf may include columns supporting the interior of the shelf. If the number of columns is n, the number of multiple areas included in the shelf may be n+1.

[0340] For example, if a rectangular shelf has six internal columns, the shelf may include seven areas (20-1, 20-2, 20-3, 20-4, 20-5, 20-6, 20-7). Wine may be placed in each of the seven areas (20-1, 20-2, 20-3, 20-4, 20-5, 20-6, 20-7). The electronic device (100) may identify in which of the seven areas (20-1, 20-2, 20-3, 20-4, 20-5, 20-6, 20-7) the wine is stored.

[0341] Referring to the embodiment (1420) of FIG. 14, the electronic device (100) can determine that the wine object is stored in the fourth area (20-4) based on the captured image.

[0342] Referring to embodiment (1430) of Fig. 14, unlike embodiment (1420), the storage direction of the wine object may be reversed. Regardless of the storage direction of the wine object, the electronic device (100) may determine that the wine object is stored in the fourth area (20-4).

[0343] FIG. 15 is a diagram illustrating an operation of identifying a wine object through an image, according to one embodiment.

[0344] Referring to the embodiment (1510) of FIG. 15, the electronic device (100) can identify the location of the wine based on the photographed image acquired in the embodiment (1010) of FIG. 10. The electronic device (100) can identify coordinate information (1510-B) of the wine object placed on the shelf object. The coordinate information (1510-B) of the wine object can identify a first vertex (1511), a second vertex (1512), a third vertex (1513), and a fourth vertex (1514).

[0345] The electronic device (100) can identify the minimum x-axis value (x1) among the entire coordinate values ​​representing the wine object based on the coordinate information (1510-B) of the wine object. The electronic device (100) can identify the maximum x-axis value (x2) among the entire coordinate values ​​representing the wine object based on the coordinate information (1510-B) of the wine object. The x-axis can be described as a preset axis. The x-axis can be parallel to a horizontal line representing the horizontal length of the shelf object.

[0346] The electronic device (100) can identify the wine position based on the horizontal length (d1) of the shelf object, the minimum x-axis value (x1) of the wine object, and the maximum x-axis value (x2) of the wine object. The electronic device (100) can calculate relative position information of the wine object in the entire shelf object. A description related thereto is described in FIG. 16.

[0347] Referring to embodiment (1520) of FIG. 15, the electronic device (100) can identify the location of the wine based on the captured image acquired in embodiment (1020) of FIG. 10. The electronic device (100) can identify coordinate information (1520-B) of the wine object placed on the shelf object. Other operations may correspond to embodiment (1510) of FIG. 15.

[0348] In embodiments (1510) and (1520), the coordinate information representing the shelf object may be partially different. However, the identified shelf location and wine location may be the same.

[0349] FIG. 16 is a diagram illustrating a table including information for specifying a wine location, according to one embodiment.

[0350] Referring to the embodiment (1610) of FIG. 16, the electronic device (100) can calculate relative position information of wine (or wine object).

[0351] xt can represent the minimum x-value of the shelf object. The minimum x-value can represent the minimum x-axis value among the coordinate information of the shelf object.

[0352] cx can represent relative position information. Relative position information can include a value indicating the relative position.

[0353] x1 can represent the minimum x-value of the wine object. The minimum x-value can represent the minimum x-axis value among the coordinate information of the wine object.

[0354] x2 can represent the maximum x-value of the wine object. The maximum x-value can represent the maximum x-axis value among the coordinate information of the wine object.

[0355] d1 can represent the horizontal length of the shelf object obtained from the photographed image.

[0356] The electronic device (100) can obtain the x-axis center value (x1+x2 / 2) of the wine object by adding the minimum x-value (x1) of the wine object and the maximum x-value (x2) of the wine object and dividing the result by 2.

[0357] The electronic device (100) can subtract the minimum x-axis value among the coordinate information of the shelf object from the x-axis center value (x1+x2 / 2) of the wine object.

[0358] The electronic device (100) can obtain relative position information (cx) by dividing the subtraction value {(x1+x2 / 2)-xt} by the horizontal length (d1) of the shelf object.

[0359] The electronic device (100) can identify the wine location based on relative location information (cx) and a second mapping table (1620). The second mapping table (1620) can represent a mapping table including relative location ranges corresponding to each of a plurality of areas included in the shelf.

[0360] If the relative position information of the wine object is greater than 0 and less than or equal to 0.14, the electronic device (100) can identify that the wine object is located in the first area.

[0361] If the relative position information of the wine object is greater than 0.14 and less than or equal to 0.29, the electronic device (100) can identify that the wine object is located in the second area.

[0362] If the relative position information of the wine object is greater than 0.29 and less than or equal to 0.43, the electronic device (100) can identify that the wine object is located in the third area.

[0363] If the relative position information of the wine object is greater than 0.43 and less than or equal to 0.57, the electronic device (100) can identify that the wine object is located in the fourth region.

[0364] If the relative position information of the wine object is greater than 0.57 and less than or equal to 0.71, the electronic device (100) can identify that the wine object is located in the fifth region.

[0365] If the relative position information of the wine object is greater than 0.71 and less than or equal to 0.86, the electronic device (100) can identify that the wine object is located in the sixth region.

[0366] If the relative position information of the wine object is greater than 0.86 and less than or equal to 1, the electronic device (100) can identify that the wine object is located in the seventh area.

[0367] According to various embodiments, the electronic device (100) may calculate the relative position using only one value, without using both the minimum x value (x1) of the wine object and the maximum x value (x2) of the wine object. Although the accuracy may be lower than when all values ​​are used, in order to reduce the calculation process, the electronic device (100) may calculate the relative position using only the minimum x value (x1) of the wine object or the maximum x value (x2) of the wine object. When using one value, the remaining values ​​may be calculated as 0.

[0368] According to various embodiments, a shelf object may be separated and a relative position may be calculated as shown in FIG. 17. The electronic device (100) may extract (or crop) an area representing the shelf object based on first coordinate information representing the shelf object. The electronic device (100) may calculate x1 and x2 values ​​based on the extracted area. xt may be calculated as 0.

[0369] FIG. 17 is a drawing for explaining an operation of cropping an image according to one embodiment.

[0370] Referring to the embodiment (1710) of FIG. 17, the electronic device (100) can extract only a portion including a shelf object and a wine object from the photographed image acquired in the embodiment (1010) of FIG. 10. The electronic device (100) can perform an image extraction (or segmentation) operation to efficiently manage data processing speed and data throughput without analyzing all portions of the photographed image. The electronic device (100) can separate an area including a shelf object and a wine object from the photographed image to acquire an extracted image. The electronic device (100) can calculate a wine location based on the extracted image. The calculation process is described in FIGS. 15 and 16.

[0371] Referring to the embodiment (1720) of FIG. 17, the electronic device (100) can extract only the portion including the shelf object and the wine object from the captured image obtained in the embodiment (1020) of FIG. 10. The description other than that may be the same as the embodiment (1710). The calculation process is described in FIGS. 15 and 16.

[0372] FIG. 18 is a drawing for explaining an operation of identifying multiple wines according to one embodiment.

[0373] Referring to the embodiment (1810) of FIG. 18, the electronic device (100) can identify multiple wine objects in a captured image. The electronic device (100) can obtain coordinate information (1811-B, 1812-B) for each of the multiple wine objects. Based on each piece of coordinate information, the electronic device (100) can identify the location of each of the multiple wine objects.

[0374] For example, the plurality of wine objects may include a first wine object and a second wine object. The electronic device (100) may obtain first coordinate information (1811-B) of the first wine object. Based on the first coordinate information (1811-B), the electronic device (100) may identify that the first wine object is located in the second area.

[0375] The electronic device (100) can obtain second coordinate information (1812-B) of the second wine object. Based on the second coordinate information (1812-B), the electronic device (100) can identify that the second wine object is located in the sixth area.

[0376] Referring to an embodiment (1820) of FIG. 18, the electronic device (100) can acquire a photographed image including a plurality of wine objects (1821, 1822, 1823, 1824, 1825, 1826, 1827) having different storage directions. The photographed image can include wine objects (1821, 1823, 1825, 1827) stored in a first direction and wine objects (1822, 1824, 1826) stored in a second direction opposite to the first direction.

[0377] The electronic device (100) can identify the location of the wine regardless of the storage direction of the wine objects. The electronic device (100) can identify that a first wine object is located in a first area, a second wine object is located in a second area, a third wine object is located in a third area, a fourth wine object is located in a fourth area, a fifth wine object is located in a fifth area, a sixth wine object is located in a sixth area, and a seventh wine object is located in a seventh area.

[0378] FIG. 19 is a diagram for explaining an operation of obtaining wine characteristic information through a wine server according to one embodiment.

[0379] Referring to the embodiment (1900) of FIG. 19, the electronic device (100) can obtain wine characteristic information. The electronic device (100) can obtain a photographed image. The electronic device (100) can extract (or obtain) a wine image including a wine object from the photographed image. The electronic device (100) can transmit the wine image to the server (200).

[0380] The server (200) can receive a wine image from the electronic device (100). The server (200) can obtain wine characteristic information based on the received wine image. The wine characteristic information can include various information related to the characteristics of the wine. The server (200) can obtain (or search) wine characteristic information corresponding to the wine image from various wine information stored in a database. The server (200) can transmit the wine characteristic information to the electronic device (100).

[0381] The electronic device (100) can receive wine characteristic information from the server (200). The electronic device (100) can store the wine characteristic information.

[0382] According to one embodiment, the electronic device (100) may transmit a photographed image instead of a wine image to the server (200). The electronic device (100) may transmit a photographed image (or an original image) instead of a wine image containing only a wine object to the server (200). The server (200) may identify a wine object in the photographed image and obtain wine characteristic information based on an image area corresponding to the wine object.

[0383] According to one embodiment, the electronic device (100) can obtain wine characteristic information using only a database within the electronic device (100) in an on-device form without using a server (200).

[0384] FIG. 20 is a drawing for explaining an operation of displaying wine information according to one embodiment.

[0385] Referring to FIG. 20, the electronic device (100) may provide a screen (2000) containing information related to wine. The screen (2000) may be a screen for providing information about a target wine.

[0386] The target wine may be at least one of a wine selected by the user, a newly registered wine, or a newly stored wine. The screen (2000) may include at least one of an area (2010) indicating the shelf location where the target wine is stored, an area (2020) indicating a wine image including the target wine, or an area (2030) indicating wine characteristic information.

[0387] The area (2010) may include a UI (2011) indicating a shelf on which the target wine is stored among a plurality of shelves included in the electronic device (100).

[0388] The region (2020) may include an image representing the target wine. The image representing the target wine may be either a preset wine sample image or a wine image extracted from a photographed image. The preset wine sample image may be an image representing the target wine. The preset wine sample image may be an image pre-stored in a database or the like. The extracted wine image may represent an image in which a portion displaying a wine object is extracted from a photographed image acquired through a camera (190) included in the electronic device (100).

[0389] The area (2030) may be an area representing information about wine characteristics acquired by the electronic device (100). The area (2030) may include various information about the target wine. The area (2030) may include the name, type, flavor, manufacturer, manufacturing date, expiration date, price, rating, etc. of the target wine.

[0390] FIG. 21 is a drawing for explaining an operation of indicating a wine position according to one embodiment.

[0391] Referring to FIG. 21, the electronic device (100) may provide a screen (2100) that displays wine storage information. The screen (2100) may include a structure of a plurality of shelves (10-1, 10-2, 10-3, 10-4, 10-5, 10-6, 10-7, 10-8, 10-9, 10-10, 10-11, 10-12) included in the electronic device (100).

[0392] The screen (2100) may include a UI (2110) indicating the location of the target wine among multiple shelves. Assume that the target wine is stored in the fourth area of ​​the ninth shelf (10-9). The electronic device (100) may display a UI (2110) indicating the storage location of the target wine.

[0393] The UI (2110) may include at least one of text information indicating the status of the target wine, an image related to the wine, or an image related to the wine label. The image may include a thumbnail image.

[0394] According to one embodiment, the UI (2110) may include text information indicating the name of the target wine. The text information included in the UI (2110) may indicate the full name or partial name of the target wine. In the case of a partial name, initial information may be indicated. For example, the UI (2110) may include only initial information (A) indicating the first letter of the full name (ABC) of the wine. Through the UI (2110) included in the screen (2100), a user can easily recognize the name of the target wine.

[0395] According to one embodiment, the UI (2110) may include text information indicating the opening status of the target wine. If the target wine is opened, the UI (2110) may include “O.” “O” may indicate “open.” If the target wine is unopened, the UI (2110) may include “N.” “N” may indicate “new.” Through the UI (2110) included in the screen (2100), a user can easily recognize whether the target wine is opened.

[0396] FIG. 22 is a drawing for explaining a situation in which multiple shelves are included in one image, according to one embodiment.

[0397] Referring to embodiment (2210) of FIG. 22, the electronic device (100) can acquire a photographed image captured while multiple shelves are open. The electronic device (100) can identify that multiple shelves are open through the photographed image. Multiple shelf objects can be identified in the photographed image.

[0398] Referring to the embodiment (2220) of FIG. 22, the electronic device (100) can identify a shelf on which wine is placed among a plurality of shelves. The electronic device (100) can identify a wine object based on a captured image and identify a shelf object on which the wine object is placed.

[0399] Referring to the embodiment (2230) of FIG. 22, the electronic device (100) may not be able to identify the shelf on which a wine object is placed among multiple shelves. If multiple shelf objects overlap, the electronic device (100) may have difficulty identifying the shelf on which the wine object is placed. If the electronic device (100) cannot identify the shelf on which the wine object is placed, the electronic device (100) may provide a notification related to this to the user. The user may push an unnecessarily open shelf using the provided notification.

[0400] The electronic device (100) can determine whether multiple shelf objects are identified in the captured image. If multiple shelf objects are identified, the electronic device (100) can separate the wine object from the background object (or area) using an AI (Artificial Intelligence) model. The electronic device (100) can identify the shelf object that is obscured by the wine object. If there are multiple shelf objects obscured by the wine object, the electronic device (100) can identify the shelf object that is positioned at the top.

[0401] The electronic device (100) can obtain coordinate information (or bounding box) corresponding to the identified shelf object. The electronic device (100) can identify the location of the wine based on the obtained coordinate information.

[0402] FIG. 23 is a drawing for explaining an operation of identifying whether wine is opened, according to one embodiment.

[0403] Referring to FIG. 23, the electronic device (100) can identify the opening status of the wine. The electronic device (100) can obtain information on the opening status of the wine based on the captured image.

[0404] The image (2310) of Fig. 23 may include an unopened target wine. The image (2310) is a sample image and may be a pre-saved image.

[0405] The image (2320) of FIG. 23 may be a wine image representing a wine object included in a photographed image.

[0406] The electronic device (100) can compare a preset area (2311) in the image (2310) with a preset area (2321) in the image (2320). The electronic device (100) can obtain information on the opening status of the wine based on the comparison result. The electronic device (100) can obtain a similarity between the preset areas (2311, 2321) and obtain information on the opening status based on the similarity. If the similarity is greater than or equal to a threshold value, the electronic device (100) can identify that the target wine is unopened. If the similarity is less than the threshold value, the electronic device (100) can identify that the target wine is open.

[0407] Opened status may not necessarily mean the wine is currently open and the contents (liquid) may spill. Opened status may indicate that the wine has been previously opened.

[0408] The preset area can represent the area around the wine cork. This preset area can be changed according to the user's settings.

[0409] When comparing the similarity of preset areas, the electronic device (100) may consider whether a label is attached, the degree of bonding of the cork, etc. For example, if there is no label surrounding the neck of the wine object, the electronic device (100) may determine that the target wine is opened. For example, if the cork of the wine object is identified to be greater than a threshold value, the electronic device (100) may determine that the target wine is opened.

[0410] The electronic device (100) can determine whether the stored wine is unopened or opened based on the wine's opening status information. If the stored wine is opened, the electronic device (100) can provide notification information regarding the opened wine. The notification information can include information guiding the user to drink the opened wine quickly.

[0411] The electronic device (100) can identify events such as the shipment of opened wine and the arrival of unopened wine. If the same type of wine is shipped and then received, the electronic device (100) can determine that the wine is preferred by the user. Once the event is identified, the electronic device (100) can update the user's preference information.

[0412] FIG. 24 is a diagram for explaining an image transformation operation for an area including a wine object, according to one embodiment.

[0413] Referring to the embodiment (2410) of FIG. 24, the electronic device (100) can obtain coordinate information (1010-B) representing a shelf object, as in the embodiment (1010) of FIG. 10.

[0414] Referring to embodiment (2420) of FIG. 24, the electronic device (100) can extract (or crop) only the area including the shelf object, as in embodiment (1710) of FIG. 17. The extracted area may have a trapezoidal shape. This is because the object included in the captured image may have a sense of perspective.

[0415] Referring to the embodiment (2430) of FIG. 24, the electronic device (100) can convert an extracted area in a trapezoidal shape into an area in a rectangular shape. The electronic device (100) can perform an image conversion operation by expanding the third vertex (1013) to the x-axis coordinate value of the first vertex (1011) and expanding the fourth vertex (1014) to the x-axis coordinate value of the second vertex (1012). In the image conversion operation, the coordinate value for the y-axis, which is perpendicular to the x-axis, can be fixed.

[0416] The electronic device (100) can identify at least one of a shelf position and a wine position based on a converted rectangular shelf image. When performing the conversion operation, the accuracy of identifying the shelf position and / or the wine position can be improved by reducing distortion of the trapezoidal shelf object due to perspective.

[0417] FIG. 25 is a drawing for explaining an operation of registering a wine image in an electronic device (100) according to one embodiment.

[0418] Referring to the embodiment (2510) of FIG. 25, the electronic device (100) may perform a wine registration process via a camera (190). The wine registration process may refer to an action of photographing wine toward the camera (190) before storing the wine on a shelf. The wine registration process may require obtaining an image including a wine object of a critical size or larger. The photographed image obtained during the wine registration process is described as a first photographed image, and the photographed image including the wine stored on the shelf is described as a second photographed image. The first photographed image may include a wine object of a first size. The second photographed image may include a wine object of a second size. The first size may be larger than the second size.

[0419] An embodiment (2520) of FIG. 25 may represent a photographed image acquired during a wine registration process. The electronic device (100) may identify a wine object from the photographed image acquired during the wine registration process. The electronic device (100) may acquire coordinate information of the wine object. The electronic device (100) may acquire size information of the wine object based on the coordinate information of the wine object. The size information of the wine object may represent a vertical length, a diagonal length, or an area representing the wine object.

[0420] If the size information of the wine object is below a threshold value, the electronic device (100) may provide a notification guiding re-capture. The notification may be provided in the form of an audio signal or an image signal. The audio signal may include a voice signal guiding re-capture. The image signal may include a UI guiding re-capture.

[0421] If the size information of the wine object is greater than a threshold value, the electronic device (100) may provide a notification indicating that the wine registration process has been completed. The notification may be provided in the form of an audio signal or an image signal. The audio signal may include a voice signal or a preset beep sound indicating that the wine registration process has been completed.

[0422] If the size information of the wine object is greater than a threshold value, the electronic device (100) can store the captured image as a sample image. The sample image may be described as a temporary image, etc.

[0423] According to one embodiment, the electronic device (100) can determine the analysis accuracy of a wine object. The analysis accuracy of the wine object can indicate the accuracy of information required to analyze information about the wine object. The analysis accuracy of the wine object can be determined based on at least one of the resolution of the captured image, the size of text (or image) included in the captured image, and the shooting angle of the captured image. The electronic device (100) can obtain the analysis accuracy of the wine object using an artificial intelligence model. If the analysis accuracy of the wine object is below a threshold value, the electronic device (100) can provide a notification guiding re-capturing.

[0424] FIG. 26 is a diagram for explaining an operation of obtaining wine characteristic information using a registered wine image according to one embodiment.

[0425] Referring to FIG. 26, steps S2610, S2615, S2620, S2630, S2640, S2650, S2660, and S2670 may correspond to steps S610, S615, S620, S630, S640, S650, S660, and S670 of FIG. 6.

[0426] When the door is opened (S2610-Y), the electronic device (100) can register a wine sample image (S2611). The electronic device (100) can obtain a first captured image including a wine object when the door is opened (S2612). The electronic device (100) can store the first captured image as a wine sample image in a database (S2613). The wine sample image may be an image of wine captured within a threshold distance from the camera (190). The wine sample image may represent an image in which the size information of the wine object in the first captured image is greater than or equal to the threshold.

[0427] According to one embodiment, the electronic device (100) can extract a wine image containing only a wine object from the first captured image. The electronic device (100) can store the extracted wine image as a wine sample image in a database.

[0428] After storing the wine sample image in the database, the electronic device (100) can acquire a second captured image (S2620). The second captured image may represent an image captured while the wine is placed on a shelf. The second captured image may be acquired after the first captured image.

[0429] The electronic device (100) can obtain shelf positions and wine positions based on the second captured image (S2630, S2640). The electronic device (100) can store wine storage information based on the second captured image (S2650).

[0430] The electronic device (100) can acquire a first photographed image corresponding to a wine object based on a database (S2661). The electronic device (100) can identify a wine object in the second photographed image and identify (or acquire or retrieve) a first photographed image corresponding to the identified wine object from the database.

[0431] The electronic device (100) can obtain wine characteristic information based on a first captured image (S2662). The first captured image may include a wine object of a larger size than the second captured image. The electronic device (100) can obtain wine characteristic information based on the first captured image, which includes a wine object of a larger size than the second captured image. Obtaining wine characteristic information based on the first captured image can provide the user with higher accuracy results.

[0432] The electronic device (100) can continuously determine whether a wine object is identified. The electronic device (100) can track the wine based on multiple images acquired in real time.

[0433] If the wine object identified in the first captured image based on the wine tracking result is identified as having moved and been stored only within the field of view range of the camera (190), the electronic device (100) can obtain wine characteristic information based on the first captured image. Since the first captured image includes a wine object of a larger size than the second captured image, the electronic device (100) can obtain wine characteristic information based on the first captured image with high analysis accuracy.

[0434] If the wine object identified in the first captured image moves out of the field of view of the camera (190) based on the wine tracking result, the electronic device (100) can compare the wine object identified in the first captured image with the wine object identified in the second captured image to determine whether they are the same. The electronic device (100) can identify the wine object in the second captured image.

[0435] If the wine object identified in the first captured image and the wine object identified in the second captured image are the same, the electronic device (100) can obtain wine characteristic information based on the first captured image.

[0436] If the wine object identified in the first captured image and the wine object identified in the second captured image are not the same, the electronic device (100) can search for a wine sample image corresponding to the wine object identified in the second captured image in the database.

[0437] The electronic device (100) can search for a wine sample image corresponding to the wine object identified in the second captured image in a database. The database can store (or include) at least one wine sample image. The electronic device (100) can search for (or identify) a wine sample image corresponding to the wine object identified in the second captured image among the at least one wine sample image stored in the database.

[0438] If there is a wine sample image corresponding to the wine object identified in the second photographed image in the database, the electronic device (100) can obtain wine characteristic information based on the wine sample image.

[0439] For example, if wine characteristic information is mapped and stored together with a wine sample image in a database, the electronic device (100) can obtain wine characteristic information mapped to the wine sample image.

[0440] If there is no wine sample image corresponding to the wine object identified in the second captured image in the database, the electronic device (100) can obtain wine characteristic information from the second captured image itself.

[0441] FIG. 27 is a diagram for explaining a wine image database according to one embodiment.

[0442] Referring to an embodiment (2700) of FIG. 27, it is assumed that the electronic device (100) stores a wine sample image (2710) in a database (111) through a wine registration process. The database (111) can store a plurality of wine sample images (2710) registered in the electronic device (100). The database (111) can store a wine sample image (2710) including a wine object larger than a threshold size. The wine sample image (2710) can be a first captured image.

[0443] With multiple wine sample images (2710) stored in the database (111), the electronic device (100) can acquire a second photographed image including wine placed on a shelf. The electronic device (100) can acquire a wine image (2720) representing a wine object from the second photographed image.

[0444] The electronic device (100) can obtain a wine sample image (2710) corresponding to a wine image (2720) extracted based on a second photographed image among a plurality of wine sample images (2710) stored in a database (111). The electronic device (100) can identify and receive the wine sample image (2710) corresponding to the wine image (2720) through the database (111).

[0445] The wine sample image (2710) may include a wine object of a larger size than the extracted wine image (2720). The electronic device (100) may transmit the wine sample image (2710) corresponding to the extracted wine image (2720) to the server (200).

[0446] The server (200) can obtain a wine sample image (2710) from the electronic device (100). The server (200) can obtain wine characteristic information based on the wine sample image (2710). Since the wine sample image (2710) includes a wine object of a larger size than the extracted wine image (2720), the server (200) can obtain accurate wine characteristic information using the wine sample image (2710). The server (200) can transmit the wine characteristic information to the electronic device (100).

[0447] The electronic device (100) can receive wine characteristic information from the server (200). The electronic device (100) can store the wine characteristic information.

[0448] The wine sample image (2710) may include a high-quality wine object. The extracted wine image (2720) may include a low-quality wine object. The larger the wine object, the better the image quality. Using a high-quality wine object can improve the recognition of wine characteristic information.

[0449] An electronic device (100) may include a feature extraction model. The electronic device (100) may input an extracted wine image (2720) into the feature extraction model. The feature extraction model may receive the wine image (2720) as input data and obtain a wine sample image (2710) corresponding to the wine image (2720) as output data. The wine image (2720) may be described as a low-quality wine image.

[0450] The electronic device (100) can obtain detailed features of a low-resolution wine image (2720) using a feature extraction model. Based on the detailed features of the low-resolution wine image (2720), the electronic device (100) can obtain a high-resolution wine image (2710) corresponding to the low-resolution image (2720) among at least one image stored in a database (111). The electronic device (100) can obtain a high-resolution wine image (2710) having a higher resolution than the low-resolution wine image (2720) using the detailed features of the low-resolution wine image (2720) obtained from the feature extraction model.

[0451] The device in which the database (111) is implemented may vary. According to one embodiment, the database (111) may be included in the memory (110) of the electronic device (100). According to one embodiment, the database (111) may be included in the server (200).

[0452] FIG. 28 is a diagram for explaining an operation of obtaining wine characteristic information using a server (200), according to one embodiment.

[0453] Steps S2810, S2815, and S2870 of FIG. 28 may correspond to steps S510, S515, and S570 of FIG. 5.

[0454] After acquiring wine storage information, the electronic device (100) can transmit the photographed image (or wine image) to the server (200) (S2851). The server (200) can receive the photographed image (or wine image) from the electronic device (100). The server (200) can acquire wine characteristic information based on the photographed image (or wine image) (S2852). The server (200) can transmit the wine characteristic information to the electronic device (100) (S2853).

[0455] The electronic device (100) can receive wine characteristic information from the server (200). The electronic device (100) can store at least one of wine storage information or wine characteristic information (S2870).

[0456] FIG. 29 is a diagram for explaining an operation of providing wine information according to one embodiment.

[0457] Referring to FIG. 29, the electronic device (100) may provide a screen including wine information. The screen including wine information may be a screen displaying at least one of wine storage information or wine characteristic information.

[0458] Referring to the embodiment (2910) of FIG. 29, the electronic device (100) can provide a screen including wine information through the display (140) of the electronic device (100).

[0459] Referring to the embodiment (2920) of FIG. 29, the electronic device (100) can provide a screen including wine information through the display of the terminal device (300).

[0460] FIG. 30 is a drawing for explaining an operation of a terminal device (300) providing wine information according to one embodiment.

[0461] Steps S3010, S3015, S3051, S3052, S3053, and S3070 of FIG. 30 may correspond to steps S2810, S2815, S2851, S2852, S2853, and S2870 of FIG. 28.

[0462] After storing wine information including wine storage information and / or wine characteristic information, the electronic device (100) can transmit the wine information to the server (200) (S3071).

[0463] The server (200) can receive wine information from the electronic device (100). The server (200) can store the wine information (S3072). While storing the wine information, the terminal device (300) can request the wine information from the server (200) (S3080).

[0464] The server (200) can receive a request for wine information from the terminal device (300). The server (200) can identify wine information in response to the wine information request. The server (200) can transmit the identified wine information to the terminal device (300) (S3081).

[0465] The terminal device (300) can receive wine information from the server (200). The terminal device (300) can provide the received wine information (S3082).

[0466] In FIG. 30, the server that acquires wine characteristic information and the server that stores the wine information are described as being the same. Depending on various embodiments, the first server that stores wine characteristic information and the second server that stores the wine information may be distinct. When the first and second servers are distinct, the terminal device (300) can request wine information from the second server. The second server can transmit the wine information to the terminal device (300) in response to the request from the terminal device (300).

[0467] FIG. 31 is a drawing for explaining an operation of providing a UI for inputting wine characteristic information according to one embodiment.

[0468] Steps S3110, S3115, S3151, S3152, S3153, and S3170 of FIG. 31 may correspond to steps S2810, S2815, S2851, S2852, S2853, and S2870 of FIG. 28.

[0469] When a photographed image (or wine image) is received from an electronic device (100), the server (200) can determine whether wine characteristic information is identified based on the received photographed image (or wine image) (S3151-2). If wine characteristic information is identified based on the photographed image (or wine image) (S3151-2-Y), the server (200) can obtain wine characteristic information (S3152) and transmit the wine characteristic information to the electronic device (100) (S3153). The electronic device (100) can store wine storage information and the wine characteristic information received from the server (200) (S3170).

[0470] If wine characteristic information cannot be identified based on the captured image (or wine image) (S3151-2-N), the server (200) may transmit a non-identification notification to the electronic device (100) (S3180). The non-identification notification may include information indicating that wine characteristic information cannot be identified based on the captured image.

[0471] The electronic device (100) may receive an unidentifiable notification from the server (200). The electronic device (100) may provide a guide UI for entering wine characteristic information (S3181). The guide UI may include at least one of information guiding the user to select information related to wine characteristic information or information guiding the user to directly enter wine characteristic information.

[0472] The electronic device (100) can obtain wine characteristic information through a guide UI (S3182). For example, the electronic device (100) can receive a first user input through the guide UI. The first user input may be an input for selecting one of a plurality of wine characteristic information items. For example, the electronic device (100) can receive a second user input through the guide UI. The second user input may include wine characteristic information directly entered by the user.

[0473] According to various embodiments, the electronic device (100) may transmit the captured image (or wine image) to an external server before performing steps S3181 and S3182. The electronic device (100) may request wine characteristic information from an external server, which is a sub-server different from the server (200), to obtain wine characteristic information. The electronic device (100) may request wine characteristic information by transmitting the captured image (or wine image) to the external server.

[0474] If wine characteristic information is identified from an external server, the external server may transmit the wine characteristic information to the electronic device (100). If wine characteristic information is not identified from the external server, the external server may transmit a non-identification notification to the electronic device (100).

[0475] When an unidentifiable notification is received from an external server, the electronic device (100) can perform steps S3181 and S3182.

[0476] FIG. 32 is a drawing for explaining an operation of a terminal device (300) providing a UI for inputting wine characteristic information, according to one embodiment.

[0477] Steps S3210, S3215, S3251, S3151-2, S3252, S3253, and S3270 of FIG. 32 may correspond to steps S3110, S3115, S3151, S3151-2, S3152, S3153, and S3170 of FIG. 31.

[0478] The server (200) can determine whether wine characteristic information is identified based on the photographed image (or wine image) received from the electronic device (100) (S3251-2).

[0479] If wine characteristic information is not identified based on the photographed image (or wine image) (S3251-2-N), the server (200) can transmit a non-identification notification to the terminal device (300).

[0480] The terminal device (300) may receive an unidentifiable notification from the server (200). The terminal device (300) may provide a guide UI for entering wine characteristic information (S3281). The guide UI may include at least one of information guiding the user to select information related to wine characteristic information or information guiding the user to directly enter wine characteristic information.

[0481] The terminal device (300) can obtain wine characteristic information through the guide UI (S3282). For example, the terminal device (300) can receive a first user input through the guide UI. The first user input may be an input for selecting one of a plurality of wine characteristic information items. For example, the terminal device (300) can receive a second user input through the guide UI. The second user input may include wine characteristic information directly entered by the user.

[0482] The terminal device (300) can transmit wine characteristic information acquired through the guide UI to the electronic device (100) (S3283).

[0483] The electronic device (100) can receive wine characteristic information from the terminal device (300). The electronic device (100) can store at least one of wine storage information and wine characteristic information (S3270).

[0484] FIG. 33 is a drawing for explaining a UI for entering wine characteristic information according to one embodiment.

[0485] Referring to FIG. 33, a screen (3300) for inputting wine characteristic information may be provided. The screen (3300) may include at least one of an area (3310) for notifying that wine characteristic information cannot be identified, an area (3320) containing an image of a target wine to be analyzed, an area (3300) for indicating that there are no search results for the target wine, an area (3340) for selecting a candidate wine, or an area (3350) for guiding direct input of wine characteristic information.

[0486] The region (3320) may be an image to be analyzed. The region (3320) may include a photographed image, a wine image, or a wine sample image.

[0487] Area (3340) may include candidate wines that are most similar, but not identical. Area (3340) may include at least one of text information and an image of the candidate wine that guides the user to select the target wine that the user wishes to keep among the candidate wines.

[0488] The area (3350) may include text information that guides the user to directly input wine characteristic information.

[0489] FIG. 34 is a drawing for explaining a UI for entering wine characteristic information according to one embodiment.

[0490] Referring to FIG. 34, a screen (3400) may be provided for a user to directly input wine characteristic information. When a user input selecting an area (3350) of FIG. 33 is received, a screen (3400) may be provided. The screen (3400) may be a screen for inputting wine characteristic information of a target wine.

[0491] A user can input at least one of the following information on the screen (3400): name, type, flavor, manufacturer, manufacturing date, expiration date, price, and rating of the wine. The input information can be saved as wine characteristic information.

[0492] FIG. 35 is a drawing for explaining a control method of an electronic device (100) according to one embodiment.

[0493] Referring to FIG. 35, a method for controlling an electronic device for storing wine may include a step of acquiring a photographed image when a door of the electronic device is opened (S3505), a step of acquiring wine storage information including a shelf position on which wine is placed and a wine position based on the photographed image (S3510), a step of acquiring wine characteristic information corresponding to the wine (S3515), and a step of storing wine information including the wine storage information and the wine characteristic information (S3520).

[0494] The step of acquiring a photographed image (S3505) can acquire a photographed image when a signal indicating that the door is opened is received.

[0495] The step of obtaining wine storage information (S3510) includes identifying a shelf object in a photographed image, obtaining a horizontal length of the shelf object, and obtaining a shelf position based on the horizontal length of the shelf object and a first mapping table stored in an electronic device, wherein the first mapping table may include a horizontal length range corresponding to each of a plurality of shelves of the electronic device.

[0496] The step of obtaining wine storage information (S3510) may obtain first coordinate information corresponding to a shelf object based on a photographed image, and obtain the length of the outer line in the downward direction of the shelf object as the horizontal length of the shelf object based on the first coordinate information.

[0497] The step of obtaining wine storage information (S3510) may include identifying a plurality of vertices corresponding to a shelf object based on first coordinate information, identifying two vertices adjacent to a camera of an electronic device among the plurality of vertices, and obtaining the length of an outline connecting the two vertices as the horizontal length of the shelf object.

[0498] The step of obtaining wine storage information (S3510) can identify a horizontal length range corresponding to a horizontal length among a plurality of horizontal length ranges included in the first mapping table, and identify a shelf corresponding to the identified horizontal length range as a shelf where wine is stored.

[0499] The step of obtaining wine storage information (S3510) can identify that the shelf location is located further away from the camera of the electronic device as the horizontal length is smaller.

[0500] The step of obtaining wine storage information (S3510) includes identifying a wine object in a photographed image, obtaining relative position information indicating a position of the wine object in an entire area of ​​a shelf based on the photographed image, and obtaining a wine position based on the relative position information and a second mapping table stored in an electronic device, wherein the second mapping table may include a relative position range corresponding to each of a plurality of areas of the shelf.

[0501] The step (S3510) of obtaining wine storage information may include obtaining first coordinate information corresponding to a shelf object based on a photographed image, obtaining the length of a downward outer line of the shelf object as the horizontal length of the shelf object based on the first coordinate information, obtaining the minimum x-axis value of the shelf object based on the first coordinate information, obtaining second coordinate information corresponding to a wine object based on the photographed image, obtaining at least one of the minimum x-axis value of the wine object or the maximum x-axis value of the wine object based on the second coordinate information, obtaining relative position information corresponding to the wine object based on at least one of the minimum x-axis value of the wine object or the maximum x-axis value of the wine object, the minimum x-axis value of the shelf object, and the horizontal length of the shelf object, and obtaining the wine position based on the relative position information.

[0502] The step of obtaining wine characteristic information (S3515) stores a wine sample image in a database through a wine registration process, and when a wine object is identified in the photographed image, obtains a wine sample image corresponding to the wine object, and obtains wine characteristic information based on the wine sample image, and the wine characteristic information may include at least one of the name, type, flavor, manufacturer, manufacturing date, expiration date, price, or rating of the wine.

[0503] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0504] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0505] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0506] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0507] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0508] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, or other operations may be added.

[0509] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In an electronic device for storing wine, door; Memory that stores instructions; camera; and comprising at least one processor; The above instructions, when executed by the at least one processor, cause the electronic device to: When the above door is opened, the captured image is acquired through the above camera, Based on the above photographed image, wine storage information including the shelf location on which the wine is placed and the wine location is obtained, Obtain wine characteristic information corresponding to the above wine, An electronic device that stores wine information including the above wine storage information and the above wine characteristic information in the memory.

2. In paragraph 1, The above electronic device, Including door sensor; The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that obtains the photographed image through the camera when a signal indicating that the door is open is received from the door sensor.

3. In paragraph 1, The above electronic device, including multiple shelves; The above instructions, when executed by the at least one processor, cause the electronic device to: Identify the shelf object in the above captured image, Obtain the horizontal length of the above shelf object, Obtain the shelf position based on the horizontal length of the shelf object and the first mapping table stored in the memory, The above first mapping table is, An electronic device comprising a plurality of horizontal length ranges corresponding to each of the plurality of shelves.

4. In paragraph 3, The above instructions, when executed by the at least one processor, cause the electronic device to: Obtain first coordinate information corresponding to the shelf object based on the above photographed image, An electronic device that obtains the length of the outermost outer line of the shelf object among the plurality of outer lines included in the photographed image as the horizontal length of the shelf object based on the first coordinate information.

5. In paragraph 4, The above instructions, when executed by the at least one processor, cause the electronic device to: Identifying a plurality of vertices corresponding to the shelf object based on the first coordinate information, Identify two vertices adjacent to the camera among the plurality of vertices, An electronic device that obtains the length of the outline connecting the two vertices as the horizontal length of the shelf object.

6. In paragraph 3, The above instructions, when executed by the at least one processor, cause the electronic device to: Identify a horizontal length range corresponding to the horizontal length among the above plurality of horizontal length ranges, An electronic device that identifies a first shelf corresponding to the identified horizontal length range as a shelf on which the wine is stored.

7. In paragraph 3, The above instructions, when executed by the at least one processor, cause the electronic device to: Identifying the shelf object based on the horizontal length of the shelf object, An electronic device wherein the shelf object is determined to be at a greater distance from the camera as the horizontal length of the shelf object decreases.

8. In paragraph 6, The above instructions, when executed by the at least one processor, cause the electronic device to: Identify the wine object in the above captured image, Based on the above photographed image, relative position information indicating the position of the wine object in the area of ​​the first shelf is obtained, Obtain the wine location based on the relative location information and the second mapping table stored in the memory, The above second mapping table is, An electronic device comprising a plurality of relative position ranges corresponding to each of a plurality of areas of the plurality of shelves.

9. In paragraph 8, The above instructions, when executed by the at least one processor, cause the electronic device to: Obtain first coordinate information corresponding to the shelf object based on the above photographed image, Based on the first coordinate information, the length of the outer line in the downward direction of the shelf object is obtained as the horizontal length of the shelf object, Obtain the first x-axis minimum value of the shelf object based on the first coordinate information, Obtain second coordinate information corresponding to the wine object based on the above photographed image, Based on the second coordinate information, at least one of the second x-axis minimum value of the wine object or the x-axis maximum value of the wine object is obtained, Obtaining the relative position information corresponding to the wine object based on at least one of the second x-axis minimum value of the wine object or the x-axis maximum value of the wine object, the first x-axis minimum value of the shelf object, and the horizontal length of the shelf object, An electronic device that obtains the wine location based on the relative location information.

10. In paragraph 8, The above instructions, when executed by the at least one processor, cause the electronic device to: The first wine sample image is stored in the database through the wine registration process, When a wine object is identified in the above-described captured image, the first wine sample image corresponding to the wine object is acquired, Obtain the wine characteristic information based on the first wine sample image, The above wine characteristic information is, An electronic device comprising at least one of the name, type, flavor, manufacturer, date of manufacture, expiration date, price, or rating of the wine.

11. In a method for controlling an electronic device for storing wine, When the door of the above electronic device is opened, a step of acquiring a photographed image; A step of obtaining wine storage information including the shelf position on which the wine is placed and the wine position based on the above photographed image; A step of obtaining wine characteristic information corresponding to the above wine; and A control method comprising: a step of storing wine information including the above wine storage information and the above wine characteristic information; 12. In paragraph 11, The step of obtaining the above photographed image is: A control method for acquiring the photographed image when a signal indicating that the door is opened is received.

13. In paragraph 11, The steps of obtaining the above wine storage information are: Identify the shelf object in the above captured image, Obtain the horizontal length of the above shelf object, Obtaining the shelf position based on the horizontal length of the shelf object and the first mapping table stored in the electronic device, The above first mapping table is, A control method comprising a plurality of horizontal length ranges corresponding to each of a plurality of shelves of the electronic device.

14. In paragraph 13, The steps of obtaining the above wine storage information are: Obtain first coordinate information corresponding to the shelf object based on the above photographed image, A control method for obtaining the length of the outer line in the lowest direction of the shelf object as the horizontal length of the shelf object based on the first coordinate information.

15. In paragraph 14, The steps of obtaining the above wine storage information are: Identifying a plurality of vertices corresponding to the shelf object based on the first coordinate information, Identifying two vertices adjacent to the camera of the electronic device among the plurality of vertices, A control method for obtaining the length of the outline connecting the two vertices as the horizontal length of the shelf object.

Images