Shelf monitoring mobile robot, shelf monitoring system, and shelf monitoring method

Abstract

One embodiment provides a shelf monitoring mobile robot comprising: a body; a driving module providing power for movement of the body; a first imaging device and a second imaging device which are provided at a first side of the body and which have different optical characteristics; a third imaging device and a fourth imaging device which are provided at a second side different from the first side of the body and which have different optical characteristics; and an electronic shelf label identification device provided on the body so as to acquire identification information of an electronic shelf label.

Description

Shelf monitoring mobile robot, shelf monitoring system, and shelf monitoring method

[0001] The present disclosure relates to a shelf monitoring mobile robot, a shelf monitoring system, and a shelf monitoring method. More specifically, it relates to a shelf monitoring mobile robot, a shelf monitoring system, and a shelf monitoring method capable of efficiently monitoring two shelves by moving between two shelves facing each other using an imaging device provided on a first side and an imaging device provided on a second side.

[0002] - Research Project Information

[0003] [Project ID] 1415188782

[0004] [Assignment No.] P0024369

[0005] [Ministry Name] Ministry of Trade, Industry and Energy

[0006] [Name of Project Management (Specialized) Agency] Korea Institute for Industrial Technology Promotion

[0007] [Research Project Name] World Class Plus Project Support

[0008] [Project Title] Development of a Cloud-based Future Retail Integrated Solution Applying Deep Learning Algorithms

[0009] [Name of Implementing Agency] SoluM Co., Ltd.

[0010] [Research Period] April 1, 2023 ~ December 31, 2026

[0011] The use of electronic labels is gradually expanding in applications such as displaying product information on items displayed in stores. Also known as electronic shelf labels or electronic tags, electronic labels connect to a server via a gateway to receive product information and display it on an electronic paper display. Their widespread adoption is attributed to the fact that they operate at low power, allowing for extended battery life, and enable changes to display information via a communication network, thereby reducing labor costs in store management.

[0012] When a new product is displayed on a shelf, a procedure is required to assign it so that the product's information is displayed on an electronic label. Generally, when a store manager reads the barcode attached to the electronic label with a terminal and then reads the product barcode and transmits it to a server, the server assigns the corresponding electronic label to the product, registers it in a database, and processes the product information by transmitting it to the electronic label so that the display is changed.

[0013] In this way, multiple products newly displayed on a shelf and multiple electronic labels assigned information of the corresponding products can be paired and placed on the shelf. Each of the multiple electronic labels includes unique identification information, and the identification information may include location information of the electronic label. A real-gram containing the display status of multiple products arranged on the shelf can be formed based on the location information of the electronic label based on the identification information of the electronic label and the product information assigned to the electronic label.

[0014] Realogram is a real-time product display that shows the state of products displayed on multiple shelves placed in the store, and can be used by the user to manage the store.

[0015] For example, by identifying which products are placed on which shelves and in which locations, the product display status within the store can be managed, and by determining the current quantity of products, the status of popular items can be assessed.

[0016] However, in order to form a real program, there is the inconvenience of having to manually identify how the products assigned to the electronic labels are currently arranged and assign that information to each electronic label one by one, which makes it difficult for the user to operate the store efficiently.

[0017] There is a need for research on shelf monitoring methods that can improve the efficiency of store management for users by monitoring products and electronic labels placed on shelves to efficiently form real-programs and provide them to users.

[0018] According to various embodiments of the present disclosure, we aim to provide a shelf monitoring mobile robot and a shelf monitoring system capable of systematically acquiring a real-time program including the display status of products displayed on a shelf and electronic shelf labels.

[0019] According to various embodiments of the present disclosure, a shelf monitoring mobile robot and a shelf monitoring system are provided that can monitor two shelves facing each other using different imaging devices.

[0020] However, the technical problems that the various embodiments of the present disclosure aim to solve are not limited to the technical problems described above, and other technical problems may exist.

[0021] One embodiment is,

[0022] A shelf monitoring mobile robot is provided, comprising: a body; a drive module providing power for movement of the body; a first imaging device and a second imaging device provided on a first side of the body and having different optical characteristics; a third imaging device and a fourth imaging device provided on a second side of the body different from the first side and having different optical characteristics; and an electronic shelf label identification device provided on the body to acquire identification information of an electronic shelf label.

[0023] In another aspect, the first side and the second side may be in opposite directions.

[0024] In another aspect, the first angle of view of the first imaging device may be larger than the second angle of view of the second imaging device, and the third angle of view of the third imaging device may be larger than the fourth angle of view of the fourth imaging device.

[0025] In another aspect, the second imaging device and the fourth imaging device may be configured to be rotatable so as to adjust the shooting angle.

[0026] In another aspect, if the upper edge and lower edge of the first shelf are not detected in the first image in which the first imaging device captures the first shelf facing the first side of the body, the operation of the second imaging device may be controlled to capture the upper region, the central region, and the lower region of the first shelf.

[0027] In another aspect, the shelf monitoring mobile robot may further include a first distance sensor provided on the first side of the body and a second distance sensor provided on the second side of the body.

[0028] In another aspect, the operation of the driving module can be controlled based on information regarding a first distance from the body to a first shelf facing the first side of the body, sensed by the first distance sensor, and information regarding a second distance from the body to a second shelf facing the second side of the body, sensed by the second distance sensor.

[0029] In another aspect, when an image of the first shelf is detected in an image captured by at least one of the first imaging device and the second imaging device, the first distance sensor may be controlled to sense a first distance from the body to the first shelf, and when an image of the second shelf is detected in an image captured by at least one of the third imaging device and the fourth imaging device, the second distance sensor may be controlled to sense a second distance from the body to the second shelf.

[0030] In another aspect, if the average value of the first distance and the second distance is below a predetermined threshold value, the body may be controlled to move closer to either the first shelf or the second shelf than to the other.

[0031] In another aspect, the body may be controlled to be closer to either the first shelf or the second shelf than to the other, and the body may be controlled to move to a position within a predetermined critical distance from either the first shelf or the second shelf.

[0032] In another aspect, the electronic shelf label identification device may be configured to acquire identification information of an adjacent electronic shelf label when the body is positioned within a predetermined threshold distance from either the first shelf or the second shelf.

[0033] In another aspect, at least one of the first imaging device and the second imaging device, or at least one of the third imaging device and the fourth imaging device, is controlled so that an image is obtained of the other one facing the body that is close to either the first shelf or the second shelf, and position information of the electronic shelf label corresponding to the captured image and the identification information of the acquired electronic shelf label can be matched and stored.

[0034] In another aspect, the electronic shelf label identification device may include a first electronic shelf label identification device provided on the first side of the body and a second electronic shelf label identification device provided on the second side of the body.

[0035] In another aspect, the electronic shelf label identification device may include a wireless communication module that acquires data of identification information of the electronic shelf label based on a wireless communication method.

[0036] One embodiment is,

[0037] A shelf monitoring mobile robot comprising a body, a first imaging device and a second imaging device provided on a first side of the body and having optical characteristics with respect to each other, a third imaging device and a fourth imaging device provided on a second side of the body different from the first side and having different optical characteristics, and an electronic shelf label identification device provided on the body to obtain identification information of an electronic shelf label, and a shelf monitoring system comprising at least one processor that performs operations to control the operation of the shelf monitoring mobile robot and a server that includes a memory storing instructions and programs for controlling the operation of the shelf monitoring mobile robot.

[0038] In another aspect, the at least one processor controls at least one of the first imaging device and the second imaging device to photograph a first shelf facing the first side of the shelf monitoring mobile robot, and controls at least one of the third imaging device and the fourth imaging device to photograph a second shelf facing the second side of the shelf monitoring mobile robot, and can generate real-gram data for at least one product provided on the first shelf and the second shelf based on the image of the first shelf photographed and the image of the second shelf photographed.

[0039] In another aspect, the at least one processor may acquire information regarding the position of the body at the time when the first shelf is photographed, acquire information regarding the position of the body at the time when the second shelf is photographed, match the image of the first shelf with the information regarding the position of the body at the time when the first shelf is photographed, and match the image of the second shelf with the information regarding the position of the body at the time when the second shelf is photographed.

[0040] In another aspect, the at least one processor controls the driving module so that the body moves to a first position closer to the first shelf than to the second shelf, controls at least one of the third imaging device and the fourth imaging device to photograph the second shelf at the first position, controls the electronic shelf label identification device to obtain identification information of a first electronic shelf label adjacent to the first position provided on the first shelf at the first position, obtains information regarding the position of the first electronic shelf label based on the identification information of the first electronic shelf label, and can match the image of the second shelf photographed at the first position with the information regarding the position of the first electronic shelf label.

[0041] One embodiment is,

[0042] A shelf monitoring method is provided that generates real-time real-gram data by photographing a first shelf provided on a first side of a shelf monitoring mobile robot and a second shelf provided on a second side different from the first side of the shelf monitoring mobile robot.

[0043] In another aspect, the shelf monitoring method may include the step of controlling at least one of a first imaging device and a second imaging device provided on a first side of the shelf monitoring mobile robot to photograph the first shelf, the step of controlling at least one of a third imaging device and a fourth imaging device provided on a second side of the shelf monitoring mobile robot to photograph the second shelf, and the step of generating real-gram data for at least one product provided on the first shelf and the second shelf based on the image of the first shelf photographed and the image of the second shelf photographed.

[0044] In another aspect, the step of controlling to photograph the first shelf may include: controlling the first imaging device to photograph the first shelf; detecting the upper edge and the lower edge of the first shelf in the image of the first shelf photographed; controlling the second imaging device to photograph the upper region, the central region, and the lower region of the first shelf when the upper edge and the lower edge of the first shelf are not detected in the image of the first shelf photographed; and generating a first combined image by connecting the images of the upper region, the central region, and the lower region of the first shelf.

[0045] In another aspect, the step of generating the realogram data may include the step of generating realogram data for at least one product arranged on the first shelf based on the first combined image.

[0046] According to various embodiments of the present disclosure, a shelf monitoring mobile robot and a shelf monitoring method can be provided, which can acquire a real-time real-gram including products displayed on a shelf and the display status of electronic shelf labels based on images acquired by the mobile robot moving inside a store and photographing a shelf.

[0047] According to various embodiments of the present disclosure, a shelf monitoring mobile robot and a shelf monitoring system can be provided that can perform a shelf monitoring task more efficiently by utilizing an imaging device provided on a first side and an imaging device provided on a second side to simultaneously monitor two shelves facing each other.

[0048] However, the effects obtainable through the various embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0049] FIG. 1 illustrates an exemplary configuration of a shelf monitoring system in one embodiment.

[0050] FIGS. 2 to 6 are intended to explain the process of a shelf monitoring mobile robot according to one embodiment moving inside a store and photographing a plurality of shelves.

[0051] FIG. 7 is a block diagram illustrating an exemplary configuration of an electronic shelf label according to one embodiment.

[0052] FIG. 8 is a block diagram illustrating an exemplary configuration of a server according to one embodiment.

[0053] FIG. 9 is a block diagram illustrating an exemplary configuration of a shelf monitoring mobile robot according to one embodiment.

[0054] FIGS. 10 and FIGS. 11 illustrate an exemplary structure of a shelf monitoring mobile robot according to one embodiment.

[0055] FIGS. 12 to 16 are intended to illustrate how rotational control occurs for an imaging device included in a shelf monitoring mobile robot according to one embodiment.

[0056] FIGS. 17 and 18 are intended to illustrate how a shelf monitoring mobile robot according to one embodiment is controlled to perform monitoring of a plurality of shelves placed inside a store.

[0057] FIG. 19 is a block diagram illustrating an exemplary configuration of an electronic device according to one embodiment.

[0058] FIG. 20 is a flowchart of a shelf monitoring method according to one embodiment.

[0059] FIG. 21 is a flowchart of a shelf monitoring method according to another embodiment.

[0060] FIG. 22 illustrates at least one marker provided on a first shelf according to one embodiment.

[0061] FIG. 23 is a flowchart of a shelf monitoring method according to another embodiment.

[0062] The present invention is capable of various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Also, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present invention is not necessarily limited to what is depicted.

[0063] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0064] FIG. 1 illustrates an exemplary configuration of a shelf monitoring system (1000) in one embodiment. FIGS. 2 to 6 are intended to explain the process of a shelf monitoring mobile robot (300) according to one embodiment moving inside a store and photographing a plurality of shelves (SF1, SF2). FIG. 7 is a block diagram illustrating an exemplary configuration of an electronic shelf label (100) according to one embodiment. FIG. 8 is a block diagram illustrating an exemplary configuration of a server (200) according to one embodiment. FIG. 9 is a block diagram illustrating an exemplary configuration of a shelf monitoring mobile robot (300) according to one embodiment. FIGS. 10 and 11 illustrate an exemplary structure of a shelf monitoring mobile robot (300) according to one embodiment. FIGS. 12 to 16 are intended to explain the rotational control of an imaging device (52) included in a shelf monitoring mobile robot (300) according to one embodiment. FIGS. 17 and 18 are intended to illustrate how a shelf monitoring mobile robot (300) according to one embodiment is controlled to perform monitoring of a plurality of shelves (SF1, SF2) placed inside a store. FIG. 19 is a block diagram illustrating an exemplary configuration of an electronic device (400) according to one embodiment. FIG. 20 is a flowchart of a shelf monitoring method (S100) according to one embodiment. FIG. 21 is a flowchart of a shelf monitoring method (S200) according to another embodiment. FIG. 22 illustrates a configuration in which at least one marker (MK1~MK4) is provided on a first shelf (SF1) according to one embodiment. FIG. 23 is a flowchart of a shelf monitoring method (S300) according to yet another embodiment.

[0065] -System(1000)

[0066] Referring to FIG. 1, a system (1000) according to one embodiment may include an electronic shelf label (100) provided on a shelf (110), a server (200), a shelf monitoring mobile robot (300), and an electronic device (400). The electronic device (400) may include a portable user terminal.

[0067] The shelf monitoring system (1000) can provide an environment in which a user can check the status of products displayed on the shelf (110) and electronic shelf labels (100) through the electronic device (400) by generating a real-time real-time program based on images acquired by a shelf monitoring mobile robot (300) moving inside a store where multiple shelves are arranged and transmitting it to an electronic device (400) used by the user.

[0068] For example, referring to FIG. 2, the shelf monitoring mobile robot (300) can efficiently perform monitoring work on the shelves by moving between the first shelf (SF1) and the second shelf (SF2) facing each other and simultaneously monitoring the two shelves using a plurality of imaging devices provided on both sides of the shelf monitoring mobile robot (300).

[0069] Additionally, referring to FIG. 3, the distance between the first shelf (SF1) and the second shelf (SF2) facing each other is relatively short, so it may be difficult for a shelf monitoring mobile robot (300) including an imaging device having a limited field of view (FOV) to monitor the two shelves simultaneously between the two shelves.

[0070] In this case, referring to FIG. 4, the system (1000) can control the shelf monitoring mobile robot (300) to photograph the other shelf (e.g., second shelf (SF2)) while in close proximity to either of the two shelves (e.g., first shelf (SF1)), thereby enabling more flexible shelf monitoring.

[0071] The electronic shelf label (100) included in the system (1000) can be linked with the server (200), and the product information displayed on the electronic shelf label (100) can be updated from the server (200). Data of the updated product information is transmitted from the server (200) to the electronic shelf label (100), and the electronic shelf label (100) can display the updated product information. In this case, the server (200) can update the product information based on user input through the electronic device (400).

[0072] For example, a plurality of electronic shelf labels (100) and a plurality of products may be provided on a shelf (110). Each of the plurality of products may be assigned to each of the plurality of electronic shelf labels (100). A first product may be assigned to any first electronic shelf label among the plurality of electronic shelf labels (100), and a second product different from the first product may be assigned to any second electronic shelf label. A first electronic shelf label and a first product may be provided adjacent to each other, and a second electronic shelf label and a second product may be provided adjacent to each other.

[0073] The server (200) can update the relevant information of multiple products assigned to multiple electronic shelf labels (100). The server (200) can update the relevant information of multiple products collectively and transmit the updated relevant information of multiple products to the multiple electronic shelf labels (100). For example, the server (200) can update the relevant information of any first product among multiple products and transmit the updated relevant information of the first product to the first electronic shelf label to which the first product is assigned. Additionally, the server (200) can update the relevant information of any other second product among multiple products and transmit the updated relevant information of the second product to the second electronic shelf label to which the second product is assigned.

[0074] Additionally, the server (200) can perform a predetermined operation to control the operation of the shelf monitoring mobile robot (300). For example, the server (200) can transmit data regarding a movement route to the shelf monitoring mobile robot (300) so that the shelf monitoring mobile robot (300) moves along the movement route within the store. Based on the data regarding the movement route received from the server (200), the shelf monitoring mobile robot (300) can move along the movement route within the store and monitor the shelf (110).

[0075] Additionally, referring to FIGS. 5 and 6, the server (200) can control the shelf monitoring mobile robot (300) to monitor the first shelf (SF1) and the second shelf S (SF2) placed facing each other within the store.

[0076] For example, a first shelf (SF1) and a second shelf (SF2) may be arranged side by side so as to be parallel to each other inside the store. The first shelf (SF1) may include a left side (A1) and a right side (A2), and the second shelf (SF2) may include a left side (A3) and a right side (A4). In this case, the right side (A2) of the first shelf (SF1) and the left side (A3) of the second shelf (SF2) may face each other.

[0077] In one embodiment, the server (200) can control the shelf monitoring mobile robot (300) to sequentially move to a plurality of positions that are closer to the first shelf (SF1) than to the second shelf (SF2) between the first shelf (SF1) and the second shelf (SF2) facing each other.

[0078] For example, referring to FIG. 5, a shelf monitoring mobile robot (300) can be controlled to acquire a first image (Im1) by photographing a plurality of products and electronic shelf labels provided on the left side (A3) of the second shelf (SF2) through a field of view (FOV) facing to the right at a position closer to the first shelf (SF1) than to the second shelf (SF2). At the same time, at that position, the shelf monitoring mobile robot (300) can be controlled to acquire identification information of the first electronic shelf label (101) provided on the first shelf (SF1) placed on the left.

[0079] Additionally, the shelf monitoring mobile robot (300) can be controlled to move a predetermined distance in a direction parallel to the length direction of the first shelf (SF1) from the above-mentioned position. Accordingly, the shelf monitoring mobile robot (300) can move to any other position closer to the first shelf (SF1) than to the second shelf (SF2). The shelf monitoring mobile robot (300) can be controlled to acquire a second image (Im2) by photographing a plurality of products and electronic shelf labels provided on the left side (A3) of the second shelf (SF2) through a field of view (FOV) facing to the right at that position. At the same time, the shelf monitoring mobile robot (300) can be controlled to acquire identification information of the second electronic shelf label (102) provided on the first shelf (SF1) placed on the left at that position.

[0080] Furthermore, for example, referring to FIG. 6, after the shelf monitoring mobile robot (300) has completed monitoring of the second shelf (SF2), the shelf monitoring mobile robot (300) can be controlled to move to any position closer to the second shelf (SF2) than to the first shelf (SF1).

[0081] The shelf monitoring mobile robot (300) can be controlled to acquire a third image (Im3) by photographing a plurality of products and electronic shelf labels provided on the right side (A2) of the first shelf (SF1) through a viewing field of view (FOV) facing left at a position closer to the second shelf (SF2) than the first shelf (SF1). At the same time, at that position, the shelf monitoring mobile robot (300) can be controlled to acquire identification information of the third electronic shelf label (103) provided on the second shelf (SF2) located on the right.

[0082] Additionally, the shelf monitoring mobile robot (300) can be controlled to move a predetermined distance in a direction parallel to the length direction of the second shelf (SF2) from the above-mentioned position. Accordingly, the shelf monitoring mobile robot (300) can move to any other position closer to the second shelf (SF2) than to the first shelf (SF1).

[0083] The shelf monitoring mobile robot (300) can be controlled to acquire a fourth image (Im4) by photographing a plurality of products and an electronic shelf label provided on the right side (A2) of the first shelf (SF1) through a field of view (FOV) facing left at the location. At the same time, the shelf monitoring mobile robot (300) at the location can be controlled to acquire identification information of the fourth electronic shelf label (104) provided on the second shelf (SF2) placed on the right.

[0084] The server (200) can perform calculations to generate a real-time real-gram based on data of an image of a shelf (110) received from a shelf monitoring mobile robot (300) and identification information of an electronic shelf label (100) provided on another shelf (110) facing the captured shelf (110).

[0085] Meanwhile, the shelf monitoring mobile robot (300) can obtain information regarding the position of the shelf monitoring mobile robot (300) by performing wireless communication with an external electronic device.

[0086] For example, a shelf monitoring mobile robot (300) transmits a beacon signal or an ultra-wideband (UWB) signal to a gateway (not shown) provided in a store, and the gateway can transmit the beacon signal or ultra-wideband signal from the shelf monitoring mobile robot (300) to a server (200).

[0087] The server (200) can obtain information regarding the location of the shelf monitoring mobile robot (300) by analyzing a beacon signal or an ultra-wideband signal received from the shelf monitoring mobile robot (300) through the gateway. The server (200) can transmit data regarding the location of the shelf monitoring mobile robot (300) to the shelf monitoring mobile robot (300).

[0088] The electronic shelf label (100), server (200), shelf monitoring mobile robot (300), and electronic device (400) can be connected to each other through a network (500). Here, the network (500) according to the embodiment may refer to a connection structure capable of exchanging information between each node, such as the electronic shelf label (100), server (200), shelf monitoring mobile robot (300), and electronic device (400).

[0089] For example, the network (500) may include, but is not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

[0090] The electronic shelf label (100) can be wirelessly connected to the server (200) through a separate gateway (not shown). In this case, multiple gateways may be placed at regular intervals within the store, and multiple gateways may be connected to the server (200) via wired connections. The electronic shelf label (100) can wirelessly communicate with adjacent gateways, and the gateways can transmit data from the electronic shelf label (100) to the server (200).

[0091] However, it is not limited to this, and the electronic shelf label (100) can be connected to an electronic device (400) via wireless communication. For example, the electronic shelf label (100) can be paired with an electronic device (400) based on Bluetooth functionality.

[0092] Below, the configuration of the electronic shelf label (100) will be described with reference to FIG. 7.

[0093] - Electronic shelf label (100)

[0094] The electronic shelf label (100) may be an electronic device mounted on a shelf (110) to display information related to a product displayed on the shelf (110). For example, the electronic shelf label (100) may display information related to the product, such as a Korean product name, an English product name, the country of origin of the product, the price of the product, raw materials, weight / calories, and discount information.

[0095] Referring to FIG. 7, the electronic shelf label (100) may include a control unit (10), a display module (11), a communication module (12), a battery (13), and a memory (14).

[0096] The control unit (10) is a device that controls the operation of the electronic shelf label (100) and may include a system-on-chip (SOC) structure including a central processing unit (CPU) and / or a graphics processing unit (GPU), etc.

[0097] The control unit (10) can control the operation of the display module (11) so that product-related information displayed on the display module (11) is updated based on a control signal received from the outside.

[0098] The display module (11) may be a device that displays information related to a product. The display module (11) may include an Electronic Paper Display (EPD) that maintains the product information display state even when power is not supplied.

[0099] The electronic paper display is suitable for an electronic shelf label (100) that needs to reduce power consumption due to its bistability, which maintains the display state for a long time even when the power supply is interrupted.

[0100] Electronic paper displays are known to include twist ball types utilizing hemispherical twist balls charged with electrostatic charge, electrophoretic displays applying electrophoresis and microcapsules, and cholesterol liquid crystal displays utilizing cholesterol liquid crystals.

[0101] However, it is not limited to this, and the display module (11) may include any one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, and a 3D display.

[0102] The communication module (12) may include various types of communication devices that enable an external device and an electronic shelf label (100) to transmit and receive data. For example, an electronic device (400) and an electronic shelf label (100) can be paired through the communication module (12), and data from the electronic device (400) can be transmitted to the electronic shelf label (100) while paired. Additionally, data transmitted from the server (200) and delivered by the gateway can be received by the communication module (12).

[0103] The battery (13) may be a device that supplies power for driving components included in the electronic shelf label (100). For example, the battery (13) may include a lithium-ion battery. However, it is not limited thereto, and the battery (13) may include various types of batteries other than lithium-ion batteries.

[0104] Additionally, the battery (13) may include a coin-shaped power supply. However, it is not limited thereto, and the battery (13) may include various types of power supplies other than a coin-shaped one.

[0105] The memory (14) may be a storage device in which data, instructions, and various programs are stored to perform operations necessary to update product-related information displayed on the display module (11). For example, the memory (14) may be a various storage device such as ROM, RAM, EPROM, flash drive, hard drive, etc.

[0106] The memory (14) and the control unit (10) can be provided on a printed circuit board and electrically connected to each other.

[0107] The configuration of the server (200) will be described below with reference to FIG. 8.

[0108] -Server(200)

[0109] The server (200) can update the relevant information of a product assigned to an electronic shelf label (100) provided on a shelf (110) and transmit the updated relevant information of the product to the electronic shelf label (100). In this case, the server (200) can update the relevant information of the product based on user input received through an electronic device (400).

[0110] Specifically, in order to update the relevant information of a product assigned to an electronic shelf label (100), the server (200) can exchange necessary data with the electronic shelf label (100) and the electronic device (400). Accordingly, the server (200) can provide the environment necessary to update the relevant information of the product.

[0111] For example, the server (200) may provide an environment in which product-related information can be updated using an electronic device (400) (e.g., a portable user terminal type electronic device, a desktop type electronic device, etc.). The server (200) may include an application, data and / or commands, etc., for the product-related information update application to operate, and may transmit data based thereon to the electronic device (300, 400).

[0112] Additionally, the server (200) can control the operation of the shelf monitoring mobile robot (300). For example, the server (200) can transmit data regarding the movement route to the shelf monitoring mobile robot (300) so that the shelf monitoring mobile robot (300) moves along the movement route within the store.

[0113] Referring to FIG. 8, the server (200) may be implemented as a predetermined computing device comprising at least one processor (21) for data processing, a memory (22) for storing applications, data and / or instructions, etc., at least one communication module (23) for exchanging data with an external device, a location information database (24) for storing location information of an electronic shelf label (100), a template database (25) for storing a template containing product-related information, a product-related information update unit (26) for updating product-related information, a mobile robot control unit (27) for controlling the operation of a shelf monitoring mobile robot (300), a real-time real-gram generation unit (28) for generating a real-time real-gram based on data acquired by the shelf monitoring mobile robot (300), an imaging device control unit (29) for controlling the operation of an imaging device (52) of the shelf monitoring mobile robot (300), a shelf detection unit (81), and a shelf distance averaging unit (82).

[0114] The processor (21) can control the overall operation of the components included in the server (200) to provide an environment in which a product-related information editing application can operate on the electronic device (400). Additionally, the processor (21) can perform a predetermined operation to remotely control the operation of the shelf monitoring mobile robot (300).

[0115] The processor (21) may be a system-on-chip (SOC) including a central processing unit (CPU) and / or a graphics processing unit (GPU), and may execute an operating system (OS) and / or application programs stored in memory (22).

[0116] The processor (21) can communicate internally with each component included in the server (200) via a system bus and may include one or more predetermined bus structures, including a local bus.

[0117] The processor (21) can be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0118] The memory (22) can store one or more of an operating system (OS), various applications, data, and commands to provide an environment necessary for controlling the shelf monitoring mobile robot (300) and for performing a method for editing product-related information.

[0119] The memory (22) may include a program area and a data area. Here, the program area according to the embodiment may be linked between an operating system (OS) and functional elements that boot the server (200), and the data area may store data generated by the use of the server (200).

[0120] In one embodiment, the memory (22) may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and may be web storage that performs storage functions on the internet. Additionally, the memory (22) may be a recording medium that is detachable from the server (200).

[0121] The communication module (23) may include various types of communication devices that enable the server (200) to transmit and receive data with an external device.

[0122] The server (200) can transmit data based on an application, data and / or commands, etc., for a product-related information editing application to operate to an electronic device (400) through a communication module (23).

[0123] Additionally, the server (200) can transmit a control signal to the monitoring mobile robot (300) to control the operation of the shelf monitoring mobile robot (300) through the communication module (23).

[0124] Additionally, for example, the communication module (23) can receive a beacon signal or a UWB signal from the shelf monitoring mobile robot (300) and transmit it to the processor (21). The processor (21) can analyze the beacon signal or the UWB signal from the shelf monitoring mobile robot (300) to generate information regarding the location of the shelf monitoring mobile robot (300).

[0125] The location information database (24) can store location information of the electronic shelf labels (100). A plurality of electronic shelf labels (100) may be provided on the shelf (110) to correspond to a plurality of products. These plurality of electronic shelf labels (100) may each be placed at a specific location on the shelf (110), and specific location information of the plurality of electronic shelf labels (100) on the shelf (110) may be stored in the location information database (24). In this case, the location information of each of the plurality of electronic shelf labels (100) may be matched with the unique identification information of each of the plurality of electronic shelf labels (100) and stored in the location information database (24).

[0126] For example, among a plurality of electronic shelf labels (100), a first electronic shelf label may be provided in the 2nd layer, 3rd column of a first shelf provided in area A among a plurality of areas. In this case, the location information of the first electronic shelf label, 'the 2nd layer, 3rd column of the first shelf in area A', may be stored in a location information database (24) as the location information of the first electronic shelf label, corresponding to the unique identification information of the first electronic shelf label.

[0127] Here, the unique identification information of the electronic shelf label (100) may include a unique identification number assigned to the electronic shelf label (100). For example, the identification information of the electronic shelf label (100) may be in the form of a string of characters, a sequence of numbers, and / or a combination thereof. However, it is not limited thereto, and the identification information of the electronic shelf label (100) may include a combination of various symbols other than characters and numbers.

[0128] However, it is not limited to this, and the unique identification information of the electronic shelf label (100) may include a unique pattern code displayed by the electronic shelf label (100).

[0129] The template database (25) may be a database that stores templates to which product-related information is applied. For example, the template database (25) may store data regarding multiple templates to which product-related information can be entered. The multiple templates may have different forms depending on user input.

[0130] For example, multiple templates can be generated based on user input from a middle manager of a store using an electronic device (400). The middle manager can create a template of the desired form through the electronic device (400).

[0131] The electronic device (400) can transmit data regarding a plurality of templates to the server (200), and the data regarding a plurality of templates can be stored in the template database (25). However, it is not limited thereto, and the template database (25) may store data regarding a plurality of previously created templates.

[0132] The product-related information update unit (26) can update the product-related information displayed on the electronic shelf label (100). For example, the product-related information update unit (26) can control the electronic shelf label (100) so that the product-related information displayed on the electronic shelf label (100) is periodically updated according to the product-related information update rule stored in the memory (22). The product-related information update rule stored in the memory (22) is predetermined by the user and can be modified at any time.

[0133] However, it is not limited to this, and the product-related information update unit (26) may also control the electronic shelf label (100) so that the product-related information displayed on the electronic shelf label (100) is updated according to user input editing the product-related information.

[0134] For example, the product-related information update unit (26) can control the electronic shelf label (100) so that information related to the product is updated by inputting information from user input into a template provided from the template database (25) and the product-related information is displayed on the electronic shelf label (100).

[0135] The mobile robot control unit (27) can remotely control the shelf monitoring mobile robot (300). For example, the mobile robot control unit (27) can control the shelf monitoring mobile robot (300) to move inside the store along a preset movement path based on data of movement path information stored in memory (22) or an external database, and to monitor the display status of products and electronic shelf labels (100) arranged on the shelf (110).

[0136] To this end, the mobile robot control unit (27) can transmit a movement control signal to the shelf monitoring mobile robot (300) via the communication module (23) to control the shelf monitoring mobile robot (300) to move along a preset movement path. At the same time, the mobile robot control unit (27) can transmit data of the preset movement path information to the shelf monitoring mobile robot (300) via the communication module (23).

[0137] Here, the movement path information can be pre-set by the user, for example, and stored in memory (22) or an external database.

[0138] The real-gram generation unit (28) can generate a real-gram in real time including the display status of products displayed on the shelf (110) and electronic shelf labels (100) based on data of images acquired by the shelf monitoring mobile robot (300) and data of information regarding the position of the shelf monitoring mobile robot (300).

[0139] In addition, the real-gram generation unit (28) may utilize data of identification information of the electronic shelf label when generating the real-gram.

[0140] For example, referring to FIG. 2, a shelf monitoring mobile robot (300) can simultaneously capture images of the first shelf (SF1) and the second shelf (SF2) between the first shelf (SF1) and the second shelf (SF2) facing each other, thereby simultaneously acquiring images of the first shelf (SF1) and the second shelf (SF2).

[0141] In the process of simultaneously acquiring images of the first shelf (SF1) and the second shelf (SF2), a beacon signal or UWB signal from the shelf monitoring mobile robot (300) is analyzed by the server (200) to generate information regarding the position of the shelf monitoring mobile robot (300).

[0142] The server (200) can generate real-time real-gram data for the first shelf (SF1) and the second shelf (SF2) by matching images of the first shelf (SF1) and the second shelf (SF2) from the shelf monitoring mobile robot (300) with information regarding the position of the shelf monitoring mobile robot (300) at the time of taking the images.

[0143] Additionally, for example, referring to FIG. 5, a shelf monitoring mobile robot (300) can obtain a first image (Im1) by photographing the second shelf (SF2) at a first position which is one of a plurality of positions that are closer to the first shelf (SF1) than the second shelf (SF2) among the first shelf (SF1) and the second shelf (SF2) facing each other.

[0144] At the same time, the shelf monitoring mobile robot (300) can obtain first identification information from the first electronic shelf label (101) provided on the first shelf (SF1) at the first position. For example, the shelf monitoring mobile robot (300) can obtain first identification information from the first electronic shelf label (101) via wireless communication.

[0145] Additionally, the shelf monitoring mobile robot (300) can acquire a second image (Im2) by photographing the second shelf (SF2) at a second location which is any other of a plurality of locations that are closer to the first shelf (SF1) than the second shelf (SF2) among the first shelf (SF1) and the second shelf (SF2) facing each other. Here, the second location may be a location spaced apart from the first location by a predetermined distance in a direction parallel to the length direction of the first shelf (SF1).

[0146] At the same time, the shelf monitoring mobile robot (300) can obtain second identification information from the second electronic shelf label (102) provided on the first shelf (SF1) at the second location. For example, the shelf monitoring mobile robot (300) can obtain second identification information from the second electronic shelf label (102) via wireless communication.

[0147] The shelf monitoring mobile robot (300) can transmit data of the first image (Im1), data of the first identification information, data of the second image (Im2), and data of the second identification information to the server (200).

[0148] The real-gram generation unit (28) can generate real-gram data including the display status of a product displayed on a first shelf (SF1) by connecting adjacent first images (Im1) and second images (Im2) based on the data of a first image (Im1), the data of first identification information, the data of a second image (Im2), and the data of second identification information transmitted from the shelf monitoring mobile robot (300).

[0149] For example, the real-gram generation unit (28) can extract first location information of the first electronic shelf label (101) from the location information database (24) based on the data of the first identification information, and extract second location information of the second electronic shelf label (102) from the location information database (24) based on the data of the second identification information.

[0150] The real-gram generation unit (28) can match the data of the first image (Im1) with the data of the first position information and can match the data of the second image (Im2) with the data of the second position information.

[0151] Accordingly, information that the first image (Im1) is an image taken at a first position of the first electronic shelf label (101) provided on the first shelf (SF1) can be stored in the memory (22). Additionally, information that the second image (Im2) is an image taken at a second position of the second electronic shelf label (102) provided on the first shelf (SF1) can be stored in the memory (22).

[0152] The real-gram generation unit (28) can generate a real-gram in real time including the display status of products and electronic shelf labels displayed on the second shelf (SF2) by connecting the first image (Im1) and the second image (Im2) corresponding to each of the first and second positions adjacent to each other.

[0153] The imaging device control unit (29) can remotely control the operation of the imaging device included in the shelf monitoring mobile robot (300).

[0154] For example, at least one imaging device included in the shelf monitoring mobile robot (300) may be configured to be rotatable so as to adjust the shooting angle. The imaging device control unit (29) may control the rotatable imaging device to rotate so that the shelf monitoring mobile robot (300) can monitor products and electronic shelf labels (100) placed on the shelf (110) at various angles.

[0155] The shelf detection unit (81) can detect an image of the shelf (110) from an image captured by the imaging device (52) of the shelf monitoring mobile robot (300).

[0156] For example, the imaging device (52) of the shelf monitoring mobile robot (300) can transmit data of an image of the surrounding environment to the server (200). The shelf detection unit (81) can detect an image of the shelf (110) from an image of the surrounding environment of the shelf monitoring mobile robot (300) through an object extraction algorithm.

[0157] When a shelf (110) is detected in an image of the surrounding environment of the shelf monitoring mobile robot (300) by the shelf detection unit (81), the distance sensor (53) of the shelf monitoring mobile robot (300) can be controlled to operate to sense the distance from the shelf monitoring mobile robot (300) to the nearby shelf (110).

[0158] The shelf distance averaging unit (82) can average the distance from the shelf monitoring mobile robot (300) to one of the two shelves facing each other and the distance from the shelf monitoring mobile robot (300) to the other shelf. The distance from the shelf monitoring mobile robot (300) to the shelf can be sensed by a distance sensor (53) included in the shelf monitoring mobile robot (300).

[0159] For example, referring to FIG. 2, the shelf monitoring mobile robot (300) may be positioned between the first shelf (SF1) and the second shelf (SF2) facing each other. In this case, the shelf monitoring mobile robot (300) may sense a first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) and a second distance from the shelf monitoring mobile robot (300) to the second shelf (SF2) using a distance sensor (53).

[0160] The shelf monitoring mobile robot (300) can transmit data regarding the first distance and the second distance to the server (200), and the shelf distance averaging unit (82) can calculate the average value of the first distance and the second distance.

[0161] In the above description, it has been explained that a server (200) according to one embodiment performs functional operations as described above; however, depending on the embodiment, at least a portion of the functional operations performed by the server (200) may be performed by an external device (e.g., a shelf monitoring mobile robot (300) or an electronic device (400)), and at least a portion of the functional operations performed by the external device may be further performed by the server (200), and various other embodiments may be possible.

[0162] Below, the configuration of the shelf monitoring mobile robot (300) will be described with reference to FIGS. 9 and FIGS. 10.

[0163] - Shelf monitoring mobile robot (300)

[0164] A shelf monitoring mobile robot (300) according to one embodiment can monitor the display status of products and electronic shelf labels (100) while moving inside a store.

[0165] For example, the shelf monitoring mobile robot (300) can acquire data of an image of a product and an electronic shelf label (100) inside the store. In this case, the shelf monitoring mobile robot (300) can also acquire data of information regarding the location where the image was acquired, along with the image data.

[0166] In this way, the shelf monitoring mobile robot (300) can simultaneously acquire an image of a product and an electronic shelf label (100) and information about the location where the image was taken, and can match the data of the image with the data of the information about the location where the image was taken.

[0167] However, it is not limited to this, and when the shelf monitoring mobile robot (300) transmits data of the captured image and data of identification information of the electronic shelf label (100) adjacent to the captured position to the server (200), the server (200) can extract location information of the electronic shelf label (100) based on the identification information of the electronic shelf label (100). Afterwards, the server (200) can match the image data received from the shelf monitoring mobile robot (300) with the extracted location information.

[0168] Referring to FIG. 9, a shelf monitoring mobile robot (300) according to one embodiment may include a power module (30), a driving module (40), a sensor module (50), a wireless communication module (60), a memory (61), a navigation module (62), and a control unit (70).

[0169] Additionally, referring to FIG. 10, the shelf monitoring mobile robot (300) may include a body (310) comprising a column portion (311) and a base (312) configured to be movable and provided at the bottom of the column portion (311). A plurality of wheels (w1, w2) that come into contact with the ground may be provided at the bottom of the base (312).

[0170] The power module (30), driving module (40), sensor module (50), wireless communication module (60), memory (61), navigation module (62), and control unit (70) can be provided inside the body (310).

[0171] The power module (30) may include a battery (31) that supplies power to the shelf monitoring mobile robot (300) and a power management device (32) that controls the supply of power from the battery (31) to other components of the shelf monitoring mobile robot (300).

[0172] The battery (31) is a source that supplies power necessary to drive the shelf monitoring mobile robot (300), and may include, for example, a lithium-ion battery.

[0173] The power management device (32) is connected to the battery (31) and can regulate the voltage and current supplied by the battery (31) to meet the power system requirements of the shelf monitoring mobile robot (300).

[0174] The drive module (40) may be a module that supports the operation of the shelf monitoring mobile robot (300) by utilizing power provided from the power module (30).

[0175] For example, the drive module (40) may include a power generation device (41) that provides power to move a plurality of wheels (w1, w2) provided on the lower part of the base (312) so that the shelf monitoring mobile robot (300) can move inside the store.

[0176] In this case, the rotational speed and rotational direction of the first wheel (w1) and the second wheel (w2) can be controlled independently. Accordingly, by controlling the rotational speed and rotational direction of the first wheel (w1) and the second wheel (w2) differently, the shelf monitoring mobile robot (300) can be controlled to change direction or rotate in place.

[0177] Additionally, the driving module (40) may include a rotating device (42) that allows the imaging device (52) to rotate relative to the base (312).

[0178] The imaging device (52) may be provided on the column portion (311) and configured to be rotatable relative to the column portion (311) so as to adjust the shooting angle. The rotation device (42) may include a rotational power transmission structure that enables the imaging device (52) to rotate relative to the column portion (311) and may provide power for rotation to the imaging device (52).

[0179] Accordingly, the shooting angle of the imaging device (52) provided in the body (310) can be adjusted, and the shelf monitoring mobile robot (300) can monitor the products and electronic shelf labels (100) provided on the shelf (110) at various angles.

[0180] The sensor module (50) may include a front detection sensor (51), an imaging device (52), a distance sensor (53), and an IMU sensor (54).

[0181] The front detection sensor (51) may include an RGB camera (l1) that captures the direction in which the shelf monitoring mobile robot (300) moves. For example, referring to FIG. 10, the RGB camera (l1) may be provided at the front of the column (311) so as to capture the front of the shelf monitoring mobile robot (300).

[0182] A visual SLAM (Simultaneous localization and mapping; SLAM) system can be implemented through an RGB camera (l1) to simultaneously map the surrounding environment and real-time location of a shelf monitoring mobile robot (300).

[0183] For example, data of an image of the surrounding environment of a shelf monitoring mobile robot (300) moving inside a store, captured by an RGB camera (l1), can be transmitted to a control unit (70) or to a server (200).

[0184] The control unit (70) or server (200) processes an image of the surrounding environment of the shelf monitoring mobile robot (300) to perform mapping of the surrounding environment of the shelf monitoring mobile robot (300), and at the same time, can estimate the real-time location of the shelf monitoring mobile robot (300) within the store.

[0185] Additionally, the front detection sensor (51) may further include a distance sensor provided in front of the shelf monitoring mobile robot (300). The distance sensor provided in front of the shelf monitoring mobile robot (300) can detect obstacles appearing in front of the shelf monitoring mobile robot (300). When an obstacle is detected by the distance sensor provided in front of the shelf monitoring mobile robot (300), the shelf monitoring mobile robot (300) can be controlled to move while avoiding the obstacle.

[0186] The imaging device (52) may be configured to photograph multiple shelves (SF1, SF2) as the shelf monitoring mobile robot (300) moves within the store. For example, the imaging device (52) may include a camera. Additionally, the imaging device (52) may include a camera capable of over-focusing.

[0187] The imaging device may include a first imaging device (c1), a second imaging device (c2), a third imaging device (c3), and a fourth imaging device (c4) provided on the body (310). For example, the first imaging device (c1) and the second imaging device (c2) may be provided on the first side of the body (310), and the third imaging device (c3) and the fourth imaging device (c4) may be provided on the second side of the body (310) that is different from the first side. In this case, the first side and the second side may be in opposite directions.

[0188] For example, referring to FIG. 10, a first imaging device (c1) and a second imaging device (c2) may be provided on the left side (x-axis direction) of the body (310) of the shelf monitoring mobile robot (300), and a third imaging device (c3) and a fourth imaging device (c4) may be provided on the right side (-x-axis direction).

[0189] Referring to FIGS. 2 and FIGS. 11, the first imaging device (c1) and the second imaging device (c2) can each photograph the first shelf (SF1) facing the first side of the body (310) through the first viewing field (Fov 1) and the second viewing field (Fov 2), and the third imaging device (c3) and the fourth imaging device (c4) can each photograph the second shelf (SF2) facing the second side of the body (310) through the third viewing field (Fov 3) and the fourth viewing field (Fov 4).

[0190] The first imaging device (c1) and the second imaging device (c2) provided on the first side of the body (310) may have different optical characteristics. For example, the first imaging device (c1) and the second imaging device (c2) may have different angles of view. In this case, the first angle of view of the first imaging device (c1) may be larger than the second angle of view of the second imaging device (c2). Accordingly, the first imaging device (c1) can capture a wider range of the surrounding environment compared to the second imaging device (c2). The first imaging device (c1), having a larger angle of view than the second imaging device (c2), may include a wide-angle camera.

[0191] Additionally, the third imaging device (c3) and the fourth imaging device (c4) provided on the second side of the body (310) may have different optical characteristics. For example, the third imaging device (c3) and the fourth imaging device (c4) may have different angles of view. In this case, the third angle of view of the third imaging device (c3) may be larger than the fourth angle of view of the fourth imaging device (c4). Accordingly, the third imaging device (c3) can capture a wider range of the surrounding environment compared to the fourth imaging device (c4). The third imaging device (c3), having a larger angle of view than the fourth imaging device (c4), may include a wide-angle camera.

[0192] The second imaging device (c2) and the fourth imaging device (c4), which have a relatively small field of view, can be configured to rotate so that the shooting angle can be adjusted.

[0193] For example, referring to FIG. 12, a first imaging device (c1) having a relatively large field of view provided on the first side of the body (310) can photograph a first shelf (SF1) facing the first side of the body (310) at a fixed angle. Here, the first shelf (SF1) to be photographed may include an upper edge (E1) and a lower edge (E2).

[0194] However, the distance from the body (310) to the first shelf (SF1) may not be sufficiently far, and in this case, the upper and lower regions of the first shelf (SF1) may not be included in the viewing field of the first imaging device (c1).

[0195] For example, referring to FIG. 13, the image (Im5) of the first shelf (SF1) captured by the first imaging device (c1) may not include the upper edge (E1) and the lower edge (E1) of the first shelf (SF1).

[0196] In this way, if the upper edge (E1) and lower edge (E1) of the first shelf (SF1) are not detected from the image (Im5) in which the first imaging device (c1) captures the first shelf (SF1), the second imaging device (c2) provided parallel to the first imaging device (c1) can be controlled to capture the upper region, the central region, and the lower region of the first shelf (SF1).

[0197] For example, referring to FIG. 14, at a first time point, the second imaging device (c2) can be controlled to rotate upward at a predetermined angle to photograph the upper area of ​​the first shelf (SF1). In this case, the second imaging device (c2) can be rotated upward sufficiently so that the upper edge (E1) of the first shelf (SF1) is included in the image of the first shelf (SF1).

[0198] Additionally, referring to FIG. 15, at a second time point different from the first time point, the second imaging device (c2) can be controlled to rotate downward at a predetermined angle from the position where the upper area of ​​the first shelf (SF1) was photographed to photograph the central area of ​​the first shelf (SF1).

[0199] Furthermore, referring to FIG. 16, at a third time point different from the first and second time points, the second imaging device (c2) can be controlled to rotate downward at a predetermined angle from the position where the central area of ​​the first shelf (SF1) was captured to capture the lower area of ​​the first shelf (SF1). In this case, the second imaging device (c2) can be rotated downward sufficiently so that the lower edge (E2) of the first shelf (SF1) is included in the image of the first shelf (SF1).

[0200] Likewise, the fourth imaging device (c4) can be controlled to rotate to photograph the upper region, central region, and lower region of the second shelf (SF2) facing the first shelf (SF1) when photographing the second shelf (SF2).

[0201] The image data of the image of the multiple shelves (SF1, SF2) captured by the imaging device (52) can be transmitted to the control unit (70) and / or server (200). The control unit (70) and / or server (200) can utilize the image data of the multiple shelves (SF1, SF2) captured from the imaging device (52) to generate a real-gram.

[0202] The distance sensor (53) can be configured to sense the distance to multiple shelves (SF1, SF2) as the shelf monitoring mobile robot (300) moves within the store.

[0203] For example, referring to FIG. 10, a first distance sensor (d1) may be provided on the left side of the shelf monitoring mobile robot (300), and a second distance sensor (d2) may be provided on the right side.

[0204] The first distance sensor (d1) can sense the distance to the shelf provided on the left side of the shelf monitoring mobile robot (300).

[0205] The second distance sensor (d2) can sense the distance to the shelf provided on the right side of the shelf monitoring mobile robot (300).

[0206] The IMU sensor (54) can be configured to measure the posture of the shelf monitoring mobile robot (300). For example, the IMU sensor (54) may include a gyroscope and an accelerometer.

[0207] The IMU sensor (54) can measure the posture of the shelf monitoring mobile robot (300) moving within the store in real time and can transmit the posture information data of the shelf monitoring mobile robot (300) to the control unit (70) and / or server (200).

[0208] The posture information of the shelf monitoring mobile robot (300) sensed by the IMU sensor (54) can be used by the control unit (70) or server (200) to more accurately estimate the real-time position of the shelf monitoring mobile robot (300) within the store when implementing a visual slam system based on data from the front detection sensor (51).

[0209] The wireless communication module (60) is a device that enables the shelf monitoring mobile robot (300) to communicate wirelessly with the electronic shelf label (100), server (200), and electronic device (400).

[0210] For example, the wireless communication module (60) may be configured to communicate wirelessly with the server (200) and the electronic device (400) via wireless communication such as LTE, 5G, etc.

[0211] Additionally, for example, the wireless communication module (60) can be configured to perform short-range wireless communication with the electronic shelf label (100).

[0212] For example, the wireless communication module (60) may include a low-power Bluetooth (Bluetooth low energy; BLE) device, an RFID (Radio frequency identification) tag, an NFC (Near field communication) tag, etc.

[0213] For example, referring to FIG. 10, the wireless communication module (60) may include a plurality of wireless signal transceivers (i1, i2) for performing short-range wireless communication with the electronic shelf label (100).

[0214] Multiple wireless signal transceivers (i1, i2) may be configured to receive BLE signals, RFID signals, etc. Additionally, multiple wireless signal transceivers (i1, i2) may be configured as NFC tags.

[0215] For example, a first wireless signal transceiver (i1) may be provided on the left side of the shelf monitoring mobile robot (300), and a second wireless signal transceiver (i2) may be provided on the right side.

[0216] The first wireless signal transceiver (i1) can wirelessly communicate with the electronic shelf label (100) displayed on the shelf provided on the left side of the shelf monitoring mobile robot (300).

[0217] Data of identification information of an electronic shelf label (100) displayed on a shelf provided on the left side of a shelf monitoring mobile robot (300) can be obtained by the first wireless signal transceiver (i1).

[0218] The second wireless signal transceiver (i2) can wirelessly communicate with the electronic shelf label (100) displayed on the shelf provided to the right of the shelf monitoring mobile robot (300).

[0219] Data of identification information of an electronic shelf label (100) displayed on a shelf provided on the right side of the shelf monitoring mobile robot (300) can be obtained by the second wireless signal transceiver (i2).

[0220] Furthermore, the wireless communication module (60) may include a communication device that transmits a beacon signal or a UWB signal. The beacon signal or UWB signal transmitted from the wireless communication module (60) may be transmitted to the server (200) through a gateway.

[0221] The memory (61) stores data that supports various functions of the shelf monitoring mobile robot (300). For example, the memory (61) can store a number of applications (application programs or applications) running on the shelf monitoring mobile robot (300), data for the operation of the shelf monitoring mobile robot (300), and commands. At least some of these applications can be downloaded from the server (200) via wireless communication.

[0222] Memory (61) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc.

[0223] The navigation module (62) can guide the shelf monitoring mobile robot (300) to move along a path inside a store where multiple shelves (SF1, SF2) are provided.

[0224] For example, referring to FIG. 17, the navigation module (62) can guide the shelf monitoring mobile robot (300) to move along a pre-set movement path (Tr1) that can monitor the first and second shelves (SF1, SF2) placed in the store based on data of pre-set movement path information from the server (200).

[0225] In this case, the navigation module (62) can match the surrounding environment of the shelf monitoring mobile robot (300) with a preset movement path (Tr1) based on mapping data of the surrounding environment of the shelf monitoring mobile robot (300). Accordingly, the navigation module (62) can guide the shelf monitoring mobile robot (300) to move along the preset movement path (Tr1) within the store where it is located.

[0226] The control unit (70) can control the overall operation of the components included in the shelf monitoring mobile robot (300) and perform data processing for a series of operations of the shelf monitoring mobile robot (300) to be described later.

[0227] The control unit (70) may be ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, or any other type of processor for performing functions.

[0228] Referring again to FIG. 2, the control unit (70) can control the shelf monitoring mobile robot (300) to move within a store where a plurality of shelves (SF1, SF2) are arranged parallel to each other and to monitor the plurality of shelves (SF1, SF2).

[0229] In this case, the control unit (70) can control the operation of the shelf monitoring mobile robot (300) based on information regarding the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) and information regarding the second distance from the shelf monitoring mobile robot (300) to the second shelf (SF2).

[0230] The first distance sensor (d1) and the second distance sensor (d2) of the shelf monitoring mobile robot (300) can be controlled to sense the first distance and the second distance, respectively. In this case, the first distance sensor (d1) and the second distance sensor (d2) can be controlled to operate periodically to sense the distance to various objects in the surrounding environment.

[0231] However, it is not limited to this, and the operation of the first distance sensor (d1) and the second distance sensor (d2) can be controlled based on the analysis results of the images captured by the first to fourth imaging devices (c1, c2, c3, c4).

[0232] For example, the control unit (70) can control the first distance sensor (d1) to sense the first distance from the body (310) to the first shelf (SF1) when an image of the first shelf (SF1) is detected in an image captured by at least one of the first imaging device (c1) and the second imaging device (c2).

[0233] For example, data of an image captured by at least one of the first imaging device (c1) and the second imaging device (c2) is transmitted to the server (200), and the shelf detection unit (81) of the server (200) can detect an image of the first shelf (SF1) in the image.

[0234] Subsequently, the imaging device control unit (29) of the server (200) can transmit a control signal to the shelf monitoring mobile robot (300) to control the operation of the first distance sensor (d1). Based on the control signal from the server (200), the control unit (70) of the shelf monitoring mobile robot (300) can control the first distance sensor (d1) to sense the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1).

[0235] Likewise, for example, the control unit (70) can control the second distance sensor (d2) to sense the second distance from the body (310) to the second shelf (SF2) when an image of the second shelf (SF2) is detected in an image captured by at least one of the third imaging device (c3) and the fourth imaging device (c4).

[0236] In this way, the on / off operation of the first distance sensor (d1) and the second distance sensor (d2) can be controlled based on the analysis result of the image taken of the shelf (110), and accordingly, power consumption due to the operation of the first distance sensor (d1) and the second distance sensor (d2) can be reduced.

[0237] Additionally, referring to FIGS. 3 to 6, the control unit (70) can control the shelf monitoring mobile robot (300) to move closer to either the first shelf (SF1) or the second shelf (SF2) than to the other when the average value of the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) and the second distance from the shelf monitoring mobile robot (300) to the second shelf (SF2) is less than or equal to a predetermined threshold value.

[0238] For example, data of information regarding a first distance sensed by a first distance sensor (d1) and data of information regarding a second distance sensed by a second distance sensor (d2) are transmitted to a server (200), and a shelf distance averaging unit (82) of the server (200) can calculate the average value of the first distance and the second distance.

[0239] Afterward, the processor (21) of the server (200) can determine whether the average value of the first distance and the second distance is below a predetermined threshold.

[0240] When the processor (21) determines that the average value of the first distance and the second distance is less than or equal to a predetermined threshold, it can transmit a drive module control signal to the shelf monitoring mobile robot (300) to move the shelf monitoring mobile robot (300) closer to either the first shelf (SF1) or the second shelf (SF1) than to the other.

[0241] In this case, based on a drive module control signal from the server (200), the control unit (70) of the shelf monitoring mobile robot (300) can control the drive module (40) to move the shelf monitoring mobile robot (300) closer to either the first shelf (SF1) or the second shelf (SF2) than to the other.

[0242] Here, the drive module control signal may be a signal for controlling the shelf monitoring mobile robot (300) to move to the shelf located closer to the shelf monitoring mobile robot (300) among the first shelf (SF1) and the second shelf (SF2).

[0243] For example, referring to FIG. 5, the shelf monitoring mobile robot (300) can be controlled to move to a first position between the first shelf (SF1) and the second shelf (SF2) that is closer to the first shelf (SF1) than to the second shelf (SF2).

[0244] At a first position closer to the first shelf (SF1) than to the second shelf (SF2), the shelf monitoring mobile robot (300) can be controlled to obtain a first image (Im1) by utilizing at least one of the third imaging device (c3) and the fourth imaging device (c4) to photograph the second shelf (SF2).

[0245] At the same time, the shelf monitoring mobile robot (300) can obtain information regarding the first position.

[0246] For example, the shelf monitoring mobile robot (300) can obtain information regarding a first location based on the analysis of a beacon signal or UWB signal by the server (200) via wireless communication.

[0247] Additionally, at the first position, the shelf monitoring mobile robot (300) can obtain identification information of the first electronic shelf label (101) adjacent to the first position among the plurality of electronic shelf labels (100) provided on the first shelf (SF1).

[0248] The control unit (70) can obtain location information of the first electronic shelf label (101) based on identification information of the first electronic shelf label (101) as information regarding the first location.

[0249] In this case, the control unit (70) can extract location information of the first electronic shelf label (101) corresponding to the identification information of the first electronic shelf label (101) from the location information database (24) of the server (200) or a database configured separately externally.

[0250] In obtaining identification information of the first electronic shelf label (101) by the shelf monitoring mobile robot (300), at least one of the first wireless signal transceiver (i1) or the first imaging device (c1) and the second imaging device (c2) may be used.

[0251] For example, the shelf monitoring mobile robot (300) can obtain identification information of the first electronic shelf label (101) by performing wireless communication with the first electronic shelf label (101) through the first wireless signal transceiver (i1).

[0252] Additionally, for example, a shelf monitoring mobile robot (300) can identify a pattern code displayed on the first electronic shelf label (101) from an image of the first electronic shelf label (101) captured by at least one of the first imaging device (c1) and the second imaging device (c2). The control unit (70) can obtain identification information of the first electronic shelf label (101) based on the pattern code identified from the image captured by at least one of the first imaging device (c1) and the second imaging device (c2).

[0253] In this regard, at least one of the second wireless signal transceiver (i1), the first imaging device (c1), and the second imaging device (c2) may be referred to as an electronic shelf label identification device.

[0254] In this manner, when the shelf monitoring mobile robot (300) is in the first position, the data of the first image (Im1) obtained by photographing the second shelf (SF2) by at least one of the third imaging device (c3) and the fourth imaging device (c4), and the data of the identification information of the first electronic shelf label (101) placed on the first shelf (SF1) obtained by the electronic shelf label identification device can be transmitted from the shelf monitoring mobile robot (300) to the server (200).

[0255] The real-gram generation unit (28) of the server (200) can match the location information of the first electronic shelf label (101) corresponding to the identification information of the first electronic shelf label (101) with the first image (Im1) and store it in memory (22).

[0256] However, it is not limited to this, and the control unit (70) of the shelf monitoring mobile robot (300) can match the location information of the first electronic shelf label (101) corresponding to the identification information of the first electronic shelf label (101) to the first image (Im1). Data in which the location information of the first electronic shelf label (101) and the first image (Im1) are matched by the control unit (70) can be transmitted to the server (200) and stored in the memory (20).

[0257] Meanwhile, as the shelf monitoring mobile robot (300) moves to a first position closer to the first shelf (SF1) than to the second shelf (SF2), the distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) can be measured by the first distance sensor (d1) provided in the column part (311).

[0258] The control unit (70) controls the drive module (40) so that the shelf monitoring mobile robot (300) moves to a first position, and can control the drive module (40) so that the shelf monitoring mobile robot (300) is located within a predetermined threshold distance from the first shelf (SF1) based on distance information from the shelf monitoring mobile robot (300) to the first shelf (SF1) measured by the first distance sensor (d1).

[0259] Here, a predetermined threshold distance may be a distance at which the first wireless signal transceiver (i1) can perform short-range wireless communication with the first electronic shelf label (101) provided on the first shelf (SF1). For example, if the first wireless signal transceiver (i1) is an NFC tag, the predetermined threshold distance may be within 10 cm.

[0260] Additionally, the predetermined threshold distance may be a distance at which an image of the first electronic shelf label (101) having sufficient resolution to identify the pattern code displayed on the first electronic shelf label (101) provided on the first shelf (SF1) can be obtained by either the first imaging device (c1) or the second imaging device (c2) of the shelf monitoring mobile robot (300).

[0261] Furthermore, referring to FIG. 5, the shelf monitoring mobile robot (300) can be controlled to move from a first position closer to the first shelf (SF1) than to the second shelf (SF2) to a second position spaced apart by a predetermined distance in a direction parallel to the length direction of the first shelf (SF1).

[0262] At the second position, the shelf monitoring mobile robot (300) can be controlled to acquire a second image (Im2) by utilizing at least one of the third imaging device (c3) and the fourth imaging device (c4) to photograph the second shelf (SF2).

[0263] At the same time, the shelf monitoring mobile robot (300) can obtain information regarding the second position.

[0264] For example, the shelf monitoring mobile robot (300) can obtain information regarding a second location based on the analysis of a beacon signal or UWB signal by the server (200) via wireless communication.

[0265] Additionally, at the second position, the shelf monitoring mobile robot (300) can obtain identification information of the second electronic shelf label (102) adjacent to the second position among the plurality of electronic shelf labels (100) provided on the second shelf (SF2).

[0266] The control unit (70) can obtain location information of the second electronic shelf label (102) based on identification information of the second electronic shelf label (102) as information regarding the second location.

[0267] The method of obtaining identification information and location information of the second electronic shelf label (102) is substantially the same as the method of obtaining identification information and location information of the first electronic shelf label (101) described above.

[0268] In this manner, when the shelf monitoring mobile robot (300) is in a second position, the data of the second image (Im2) obtained by photographing the second shelf (SF2) by at least one of the third imaging device (c3) and the fourth imaging device (c4), and the data of the identification information of the second electronic shelf label (102) placed on the first shelf (SF1) obtained by the electronic shelf label identification device can be transmitted from the shelf monitoring mobile robot (300) to the server (200).

[0269] The real-gram generation unit (28) of the server (200) can match the location information of the second electronic shelf label (102) corresponding to the identification information of the second electronic shelf label (102) with the second image (Im2) and store it in memory (22).

[0270] However, it is not limited to this, and the control unit (70) of the shelf monitoring mobile robot (300) can match the location information of the second electronic shelf label (102) corresponding to the identification information of the second electronic shelf label (102) to the second image (Im2). Data in which the location information of the second electronic shelf label (102) and the second image (Im2) are matched by the control unit (70) can be transmitted to the server (200) and stored in the memory (20).

[0271] Meanwhile, the control unit (70) can determine a predetermined separation distance between the first position and the second position based on the viewing angle information of either the third imaging device (c3) or the fourth imaging device (c4) that photographs the second shelf (SF2) and the distance information from the shelf monitoring mobile robot (300) to the second shelf (SF2) measured by the second distance sensor (d2) when controlling the drive module (40) so that the shelf monitoring mobile robot (300) moves from the first position to the second position.

[0272] For example, referring to FIG. 18, the distance (D) from the shelf monitoring mobile robot (300) to the second shelf (SF2) can be measured by the second distance sensor (d2) at the first position.

[0273] The control unit (70) can determine a predetermined separation distance between the first position and the second position as twice the product of the distance (D) from the shelf monitoring mobile robot (300) to the second shelf (SF1) and the tangent value of half the angle of view (θ1) of the third imaging device (c1) or the fourth imaging device (c4) that photographs the second shelf (SF2).

[0274] Additionally, with reference to FIG. 6, the shelf monitoring mobile robot (300) can acquire multiple images (Im3, Im4) of the first shelf (SF1) by utilizing the first imaging device (c1) or the second imaging device (c2) at multiple locations closer to the second shelf (SF2) than to the first shelf (SF1).

[0275] Additionally, the shelf monitoring mobile robot (300) can sequentially acquire identification information for a plurality of electronic shelf labels (103, 104) adjacent to each of the plurality of positions of the shelf monitoring mobile robot (300) that photograph the first shelf (SF1) among a plurality of electronic shelf labels (100) provided on the second shelf (SF2) at a plurality of positions closer to the second shelf (SF2) than to the first shelf (SF1).

[0276] Furthermore, data in which each of the location information of a plurality of electronic shelf labels (103, 104) corresponding to the identification information of a plurality of electronic shelf labels (103, 104) obtained at a plurality of locations closer to the second shelf (SF2) than the first shelf (SF1) is matched to a plurality of images (Im3, Im4) of the first shelf (SF1) obtained at a plurality of locations closer to the second shelf (SF2) than the first shelf (SF1) can be stored in the memory (20) of the server (200).

[0277] Meanwhile, the real-gram generation unit (28) of the server (200) can generate real-gram data including the display status of products displayed on the second shelf (SF2) by connecting adjacent first images (Im1) and second images (Im2) based on data generated by matching information regarding the first image (Im1) and the first location and matching information regarding the second image (Im2) and the second location.

[0278] In a similar manner, the real-gram generation unit (28) of the server (200) may generate real-gram data including the display status of products displayed on the first shelf (SF1).

[0279] The real-gram generation unit (28) of the server (200) can connect the first image (Im1) and the second image (Im2) by utilizing a feature extraction algorithm to extract the overlapping area between the first image (Im1) and the second image (Im2), and connect the first image (Im1) and the second image (Im2) so that the overlapping area does not overlap.

[0280] In another embodiment, when the processor (21) of the server (200) determines that the average value of the first distance and the second distance exceeds a predetermined threshold, it can transmit a drive module control signal to the shelf monitoring mobile robot (300) to move the shelf monitoring mobile robot (300) to an intermediate point between the first shelf (SF1) and the second shelf (SF2).

[0281] In this case, the control unit (70) of the shelf monitoring mobile robot (300) can control the drive module (40) based on a drive module control signal from the server (200) so that the shelf monitoring mobile robot (300) moves to an intermediate point between the first shelf (SF1) and the second shelf (SF2).

[0282] In the process of the shelf monitoring mobile robot (300) moving to the midpoint between the first shelf (SF1) and the second shelf (SF2), the drive module (40) can be controlled so that the shelf monitoring mobile robot (300) moves until the first distance sensed by the first distance sensor (d1) and the second distance sensed by the second distance sensor (d2) become equal.

[0283] The configuration of the electronic device (400) will be described below with reference to FIG. 19.

[0284] -Electronic device (400)

[0285] The electronic device (400) may be a device installed with a shelf monitoring application and / or a product-related information editing application. The electronic device (400) may include a portable user terminal.

[0286] For example, the user of the electronic device (400) may be a store employee or a customer of the store.

[0287] The electronic device (400) can be implemented, for example, as a computer or portable terminal that can access the server (200) through a network (500).

[0288] Here, the computer may include, for example, a laptop, desktop, laptop, VR HMD (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.) equipped with a web browser.

[0289] A portable terminal is a wireless communication device that ensures portability and mobility, and may include, for example, smartphones, tablet PCs, and wearable devices, as well as various devices equipped with communication modules such as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, ultrasonic, infrared, Wi-Fi, and Li-Fi.

[0290] Referring to FIG. 19, the electronic device (400) may include a processor (91), memory (92), a communication module (93), an input module (94), and a display module (95).

[0291] Various components included in the electronic device (400) can be designed to be contained within the housing of the electronic device (400).

[0292] In an embodiment, the processor (91) can control the overall operation of the components included in the electronic device (400) through the shelf monitoring application and / or product-related information editing application of the memory (92) to provide a shelf monitoring environment and / or an environment for editing product-related information.

[0293] For example, the processor (91) can control the operation of the communication module (93) and the input module (94) so ​​that a signal according to user input received through the input module (94) can be transmitted to the electronic shelf label (100) through the communication module (93).

[0294] The processor (91) may include a central processing unit (CPU) and / or a graphics processing unit (GPU). Additionally, the processor (91) may include at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0295] The memory (92) can store commands and data that can be used to create a shelf monitoring environment and / or an environment for editing related information of products.

[0296] In memory (92), a shelf monitoring application and / or an application for editing information related to the product may be stored.

[0297] The shelf monitoring application can provide a user interface that controls the operation of the shelf monitoring mobile robot (300) or provides real-program information based on data acquired by the shelf monitor mobile robot (300).

[0298] A product related information editing application can provide various types of user interfaces to provide an environment for editing product related information.

[0299] The memory (92) may include at least one non-transient computer-readable storage medium and a transient computer-readable storage medium. For example, the memory (92) may be various storage devices such as ROM, EPROM, flash drive, hard drive, etc. Additionally, the memory (92) may include web storage that performs data storage functions on the internet.

[0300] The communication module (93) may include various types of communication devices capable of transmitting and receiving data with an external device. For example, the communication module (93) may transmit and receive data with the electronic shelf label (100) and / or server (200) via a wireless network.

[0301] The communication module (93) may be configured to transmit and receive data with an external device via Bluetooth. For example, the communication module (93) may receive a pairing signal from the electronic shelf label (100) and be paired with the electronic shelf label (100) via Bluetooth.

[0302] The input module (94) may be configured to receive various forms of user input from a user using the electronic device (400). For example, the input module (94) may include a touch screen that receives touch input from the user.

[0303] When the input module (94) is implemented as a touch screen, the input module (94) may be formed by being integrally combined with the display module (95). However, it is not limited thereto, and the input module (95) may further include a keyboard capable of receiving user input in the form of characters.

[0304] The display module (95) can display content related to a shelf monitoring application and / or a product-related information update application included in the memory (92).

[0305] For example, the display module (95) may include a display device that displays various types of user interface images to provide a control environment for a shelf monitoring mobile robot (300) for shelf monitoring or to provide an environment for updating relevant information about products.

[0306] The display module (95) may include any one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, and a 3D display.

[0307] Hereinafter, a shelf monitoring method (S100, S200, S300) will be described with reference to FIGS. 20 to 23.

[0308] - Shelf monitoring method (S100, S200, S300)

[0309] A shelf monitoring method (S100, S200, S300) can be performed by controlling a shelf monitoring mobile robot (300) by a processor (21) of a server (200) according to one embodiment.

[0310] However, it is not limited to this, and at least part of the method (S100, S200, S300) may be performed by the control unit (70) of the shelf monitoring mobile robot (300) or the processor (91) of the electronic device (400), and other part may be performed by the processor (21) of the server (200).

[0311] For example, at least one of the processor (21) of the server (200), the control unit (70) of the shelf monitoring mobile robot (300), and the processor (91) of the electronic device (400) can perform a shelf monitoring method (S100, S200, S300) by controlling the shelf monitoring mobile robot (300) to monitor shelves in a store by executing at least one command stored in the memory (22) of the server (200), the memory (61) of the shelf monitoring mobile robot (300), or the memory (92) of the electronic device (400).

[0312] In the following description, the processor (21) of the server (200) controls the shelf monitoring mobile robot (300) to perform the method (S100, S200, S300).

[0313] Referring to FIG. 20, a shelf monitoring method (S100) according to one embodiment may include the step (S101) of controlling at least one of a first imaging device (c1) and a second imaging device (c2) to photograph a first shelf (SF1) facing the first side of a shelf monitoring mobile robot (300); the step (S103) of controlling at least one of a third imaging device (c3) and a fourth imaging device (c4) to photograph a second shelf (SF2) facing the second side of a shelf monitoring mobile robot (300); and the step (S105) of generating real-gram data for at least one product provided on the first shelf (SF1) and the second shelf (SF2) based on the image of the first shelf (SF1) and the image of the second shelf (SF2).

[0314] In step (S101), the processor (21) of the server (200) can control at least one of the first imaging device (c1) and the second imaging device (c2) of the shelf monitoring mobile robot (300) to photograph the first shelf (SF1) facing the first side of the shelf monitoring mobile robot (300).

[0315] For example, referring to FIG. 2, when a shelf monitoring mobile robot (300) is positioned between a first shelf (SF1) and a second shelf (SF2) facing each other, the processor (21) of the server (200) can control at least one of a first imaging device (c1) and a second imaging device (c2) provided on the first side of the shelf monitoring mobile robot (300) to photograph the first shelf (SF1).

[0316] The shelf monitoring mobile robot (300) can obtain multiple images of the first shelf (SF1) by moving between the first shelf (SF1) and the second shelf (SF2) in a direction perpendicular to the direction in which the first shelf (SF1) and the second shelf (SF2) face each other.

[0317] In this case, the operation of the shelf monitoring mobile robot (300) can be controlled based on information regarding the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) and information regarding the second distance from the shelf monitoring mobile robot (300) to the second shelf (SF2).

[0318] For example, if the average value of the first distance and the second distance is below a predetermined threshold and the second distance is shorter than the first distance, the shelf monitoring mobile robot (300) can be controlled to move to a position closer to the second shelf (SF2) than to the first shelf (SF1), as shown in FIG. 6.

[0319] Additionally, with reference to FIG. 6, the shelf monitoring mobile robot (300) can sequentially photograph the first shelf (SF1) at multiple locations closer to the second shelf (SF2) than the first shelf (SF1) to obtain multiple images of the first shelf (SF1). Furthermore, the shelf monitoring mobile robot (300) can obtain identification information of multiple electronic shelf labels (103, 104) located close to the corresponding location among the multiple electronic shelf labels (100) provided on the second shelf (SF2) at multiple locations closer to the second shelf (SF2) than the first shelf (SF1).

[0320] Additionally, for example, if the average value of the first distance and the second distance exceeds a predetermined threshold, the shelf monitoring mobile robot (300) may be controlled to move to the center area of ​​the first shelf (SF1) and the second shelf (SF2). The shelf monitoring mobile robot (300) may be controlled to photograph the first shelf (SF1) using at least one of the first imaging device (c1) and the second imaging device (c2) in the center area of ​​the first shelf (SF1) and the second shelf (SF2).

[0321] In step (S103), the processor (21) of the server (200) can control at least one of the third imaging device (c3) and the fourth imaging device (c4) of the shelf monitoring mobile robot (300) to photograph the second shelf (SF2) facing the second side of the shelf monitoring mobile robot (300).

[0322] For example, referring to FIG. 2, when a shelf monitoring mobile robot (300) is positioned between a first shelf (SF1) and a second shelf (SF2) facing each other, the processor (21) of the server (200) can control at least one of a third imaging device (c3) and a fourth imaging device (c4) provided on the second side of the shelf monitoring mobile robot (300) to photograph the second shelf (SF2).

[0323] The shelf monitoring mobile robot (300) can obtain multiple images of the second shelf (SF2) by moving between the first shelf (SF1) and the second shelf (SF2) in a direction perpendicular to the direction in which the first shelf (SF1) and the second shelf (SF2) face each other.

[0324] In this case, if the average value of the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1) and the second distance from the shelf monitoring mobile robot (300) to the second shelf (SF2) is less than or equal to a predetermined threshold value and the first distance is shorter than the second distance, the shelf monitoring mobile robot (300) can be controlled to move to a position closer to the first shelf (SF1) than to the second shelf (SF2), as shown in FIG. 5.

[0325] Additionally, with reference to FIG. 5, the shelf monitoring mobile robot (300) can sequentially photograph the second shelf (SF2) at multiple locations closer to the first shelf (SF1) than the second shelf (SF2) to obtain multiple images of the second shelf (SF1). Furthermore, the shelf monitoring mobile robot (300) can obtain identification information of multiple electronic shelf labels (101, 102) located close to the corresponding location among the multiple electronic shelf labels (100) provided on the first shelf (SF1) at multiple locations closer to the first shelf (SF1) than the second shelf (SF2).

[0326] Additionally, for example, if the average value of the first distance and the second distance exceeds a predetermined threshold, the shelf monitoring mobile robot (300) may be controlled to move to the center area of ​​the first shelf (SF1) and the second shelf (SF2). The shelf monitoring mobile robot (300) may be controlled to photograph the second shelf (SF2) using at least one of the third imaging device (c3) and the fourth imaging device (c4) in the center area of ​​the first shelf (SF1) and the second shelf (SF2).

[0327] Step (S103) can be performed substantially simultaneously with Step (S101). Accordingly, the operation of the shelf monitoring mobile robot (300) can be controlled so that imaging of the first shelf (SF1) using at least one of the first imaging device (c1) and the second imaging device (c2) and imaging of the second shelf (SF2) using at least one of the third imaging device (c3) and the fourth imaging device (c4) are performed simultaneously.

[0328] In step (S105), the real-gram generation unit (28) of the server (200) can generate real-gram data for at least one product on the first shelf (SF1) and at least one product on the second shelf (SF2) based on an image of the first shelf (SF1) and an image of the second shelf (SF2).

[0329] For example, the real-gram generation unit (28) can generate first real-gram data including the display state of at least one product arranged on the first shelf (SF1) based on a plurality of images of the first shelf (SF1) acquired by the shelf monitoring mobile robot (300) in step (S101).

[0330] Additionally, for example, the real-gram generation unit (28) can generate second real-gram data including the display state of at least one product arranged on the second shelf (SF2) based on a plurality of images of the second shelf (SF2) acquired by the shelf monitoring mobile robot (300) in step (S103).

[0331] Referring to FIG. 21, a shelf monitoring method (S200) according to another embodiment may include the steps of: controlling a first imaging device (c1) to photograph a first shelf (SF1) facing a first side of a shelf monitoring mobile robot (300) (S201); determining whether an upper edge (E1) and a lower edge (E2) of the first shelf (SF1) are detected in the image of the first shelf (SF1) (S203); controlling a second imaging device (c2) to photograph an upper area, a central area, and a lower area of ​​the first shelf (SF1) (S205); generating a first combined image by connecting the images of the upper area, a central area, and a lower area of ​​the first shelf (SF1) (S207); and generating real-gram data for at least one product provided on the first shelf (SF1) based on the image of the first shelf (SF1) (S209).

[0332] In step (S201), the processor (21) of the server (200) can control the first imaging device (c1) of the shelf monitoring mobile robot (300) to photograph the first shelf (SF1) facing the first side of the shelf monitoring mobile robot (300).

[0333] The first imaging device (c1) may be configured, for example, to have its angle fixed on the first side of the shelf monitoring mobile robot (300). However, it is not limited thereto, and the first imaging device (c1) may be configured to be rotatable on the first side of the shelf monitoring mobile robot (300) so that the shooting angle can be adjusted.

[0334] For example, referring to FIG. 2, when a shelf monitoring mobile robot (300) is positioned between a first shelf (SF1) and a second shelf (SF2) facing each other, the processor (21) of the server (200) can control a first imaging device (c1) provided on the first side of the shelf monitoring mobile robot (300) to photograph the first shelf (SF1).

[0335] The shelf monitoring mobile robot (300) can obtain multiple images of the first shelf (SF1) by moving between the first shelf (SF1) and the second shelf (SF2) in a direction perpendicular to the direction in which the first shelf (SF1) and the second shelf (SF2) face each other.

[0336] In this case, the drive of the shelf monitoring mobile robot (300) is controlled based on information regarding the first distance from the shelf monitoring mobile robot (300) to the first shelf (SF1), which may be the same as in step (S101).

[0337] In step (S203), the shelf detection unit (81) of the server (200) can detect the upper edge (E1) and lower edge (E2) of the first shelf (SF1) by utilizing an object extraction algorithm in an image of the first shelf (SF1) captured by the first imaging device (c1) of the shelf monitoring mobile robot (300).

[0338] However, it is not limited thereto, and the shelf detection unit (81) can determine whether the upper edge (E1) and lower edge (E2) of the first shelf (SF1) are included in the image of the first shelf (SF1) based on at least one marker (MK1~MK4) provided on the first shelf (SF1) detected from the image of the first shelf (SF1) taken by the first imaging device (c1).

[0339] Here, at least one marker (MK1 to MK4) may be formed to have a specific pattern and may correspond to the relevant data of the shelf (110). For example, at least one marker (MK1 to MK4) may include a QR code, but is not limited thereto.

[0340] For example, referring to FIG. 22, a first marker (MK1) and a second marker (MK2) may be provided at each end of the upper edge (E1) of the first shelf (SF1). Additionally, a third marker (MK3) and a fourth marker (MK4) may be provided at each end of the lower edge (E2) of the first shelf (SF1).

[0341] The first marker (MK1) and the second marker (MK2) may include information regarding the location of the upper edge (E1), and the third marker (MK3) and the fourth marker (MK4) may include information regarding the location of the lower edge (E2).

[0342] When the shelf detection unit (81) detects all of the first marker (MK1), second marker (MK2), third marker (MK3), and fourth marker (MK4) from the image of the first shelf (SF1), it can be determined that the image of the first shelf (SF1) includes both the upper edge (E1) and the lower edge (E2) of the first shelf (SF1).

[0343] In contrast, if the shelf detection unit (81) does not detect any of the first marker (MK1), second marker (MK2), third marker (MK3), and fourth marker (MK4) from the image of the first shelf (SF1), it can be determined that the upper edge (E1) and lower edge (E2) of the first shelf (SF1) are not included in the image of the first shelf (SF1).

[0344] In step (S203), if the upper edge (E1) and lower edge (E2) of the first shelf (SF1) are detected from the image of the first shelf (SF1), the process proceeds to step (S209) and real-gram data for at least one product provided on the first shelf (SF1) can be generated based on the image of the first shelf (SF1).

[0345] In contrast, if, in step (S203), either the upper edge (E1) or the lower edge (E2) of the first shelf (SF1) is not detected from the image of the first shelf (SF1), the process may proceed to step (S205).

[0346] In step (S205), the processor (21) of the server (200) can be controlled so that the second imaging device (c2) photographs the upper area, the central area, and the lower area of ​​the first shelf (SF1). For example, the second imaging device (c2) may be arranged side by side with the first imaging device (c1) and configured to be rotatable so as to adjust the shooting angle.

[0347] For example, as illustrated in FIGS. 14 to 16, the second imaging device (c2) can be controlled to rotate so that the shooting angle of the second imaging device (c2) is adjusted, and accordingly, the second imaging device (c2) can be controlled to shoot the upper area, the central area, and the lower area of ​​the first shelf (SF1).

[0348] In this case, the second imaging device (c2) can be controlled to rotate sufficiently upward or downward so that the upper edge (E1) and lower edge (E2) of the first shelf (SF1) are included in the image of the first shelf (SF1).

[0349] Data of the image of the first shelf (SF1) captured by the second imaging device (c2) can be transmitted to the control unit (70) and / or server (200). The control unit (70) and / or server (200) can utilize the data of the image of the first shelf (SF1) captured from the second imaging device (c2) to generate a real-gram.

[0350] In step (S207), the processor (21) of the server (200) can generate a first combined image by connecting images of the upper region, the central region, and the lower region of the first shelf (SF1). The first combined image may include the upper edge (E1) and the lower edge (E2) of the first shelf (SF1).

[0351] In step (S209), the processor (21) of the server (200) can generate real-gram data for at least one product placed on the first shelf (SF1) based on an image of the first shelf (SF1).

[0352] For example, the processor (21) of the server (200) can generate real-gram data for at least one product provided on the first shelf (SF1) based on a first combined image in which images of the upper area, central area, and lower area of ​​the first shelf (SF1) are connected.

[0353] Referring to FIG. 23, a shelf monitoring method (S300) according to another embodiment may include the steps of: controlling a third imaging device (c3) to photograph a second shelf (SF2) facing a second side of a shelf monitoring mobile robot (300) (S301); determining whether the upper edge and lower edge of the second shelf (SF2) are detected in the image of the second shelf (SF2) (S303); controlling a fourth imaging device (c4) to photograph the upper area, central area, and lower area of ​​the second shelf (SF2) (S305); generating a second combined image by connecting the images of the upper area, central area, and lower area of ​​the second shelf (SF2) (S307); and generating real-gram data for at least one product provided on the second shelf (SF2) based on the image of the second shelf (SF2) (S309).

[0354] The content of the method (S300) is substantially the same as the content of the method (S200), except that instead of photographing the first shelf (SF1) facing the first side of the shelf monitoring mobile robot (300) using the first imaging device (c1) and the second imaging device (c2), the second shelf (SF2) facing the second side of the shelf monitoring mobile robot (300) is photographed using the third imaging device (c3) and the fourth imaging device (c4). Therefore, the description of the method (S300) is replaced with the description of the method (S200).

[0355] Meanwhile, method (S200) and method (S300) can be performed simultaneously. For example, the processor (21) of the server (200) can control the first imaging device (c1) and the second imaging device (c2) to photograph the first shelf (SF1) facing the first side of the shelf monitoring mobile robot (300), and at the same time, control the third imaging device (c3) and the fourth imaging device (c4) to photograph the second shelf (SF2) facing the second side of the shelf monitoring mobile robot (300).

[0356] The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present invention, and vice versa.

[0357] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the invention.

[0358] Furthermore, although the detailed description of the present invention has been explained with reference to preferred embodiments of the invention, those skilled in the art or those with ordinary knowledge in the relevant technical field will understand that various modifications and changes can be made to the invention without departing from the spirit and technical scope of the invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

[0359] The present invention has industrial applicability in that it can improve the efficiency of store management by utilizing an imaging device provided on the first side and an imaging device provided on the second side to simultaneously monitor two shelves facing each other inside a store and acquire a real-time program including the display status of products displayed on the shelves and electronic shelf labels.

Claims

1. Body; A drive module that provides power for the movement of the above body; A first imaging device and a second imaging device having different optical characteristics, provided on the first side of the body; A third imaging device and a fourth imaging device having different optical characteristics, provided on a second side different from the first side of the body; and A shelf monitoring mobile robot comprising: an electronic shelf label identification device provided in the body to acquire identification information of an electronic shelf label.

2. In Paragraph 1, A shelf monitoring mobile robot in which the first side and the second side are in opposite directions.

3. In Paragraph 1, A shelf monitoring mobile robot, wherein the first field of view of the first imaging device is larger than the second field of view of the second imaging device, and the third field of view of the third imaging device is larger than the fourth field of view of the fourth imaging device.

4. In Paragraph 3, A shelf monitoring mobile robot in which the second imaging device and the fourth imaging device are configured to be rotatable so as to adjust the shooting angle.

5. In Paragraph 4, If the upper edge and lower edge of the first shelf are not detected in the first image captured by the first imaging device of the first shelf facing the first side of the body, A shelf monitoring mobile robot in which the operation of the second imaging device is controlled to photograph the upper area, central area, and lower area of ​​the first shelf.

6. In Paragraph 1, A first distance sensor provided on the first side of the body; and A second distance sensor provided on the second side of the body; further comprising A shelf monitoring mobile robot in which the operation of the drive module is controlled based on information regarding a first distance from the body to a first shelf facing the first side of the body, sensed by the first distance sensor, and information regarding a second distance from the body to a second shelf facing the second side of the body, sensed by the second distance sensor.

7. In Paragraph 6, When the average value of the first distance and the second distance is less than or equal to a predetermined threshold value, the body is controlled to move closer to either the first shelf or the second shelf than to the other, and A shelf monitoring mobile robot, wherein the body is controlled to be closer to either the first shelf or the second shelf than to the other, and the body is controlled to move to a position within a predetermined threshold distance from either the first shelf or the second shelf.

8. In Paragraph 7, The above electronic shelf label identification device is a shelf monitoring mobile robot configured to acquire identification information of an adjacent electronic shelf label while the body is positioned within a predetermined threshold distance from either the first shelf or the second shelf.

9. In Paragraph 8, At least one of the first imaging device and the second imaging device, or at least one of the third imaging device and the fourth imaging device, is controlled so that an image is obtained of the other one facing the body which is adjacent to either the first shelf or the second shelf. A shelf monitoring mobile robot in which location information of the electronic shelf label corresponding to the captured image and the acquired identification information of the electronic shelf label is matched and stored.

10. In Paragraph 1, The above electronic shelf label identification device is a shelf monitoring mobile robot comprising a wireless communication module that acquires data of identification information of the electronic shelf label based on a wireless communication method.

11. A shelf monitoring mobile robot comprising: a body; a first imaging device and a second imaging device provided on a first side of the body and having optical characteristics of each other; a third imaging device and a fourth imaging device provided on a second side of the body different from the first side and having different optical characteristics; and an electronic shelf label identification device provided on the body to acquire identification information of an electronic shelf label; and A server comprising at least one processor for performing operations to control the operation of the shelf monitoring mobile robot and a memory storing instructions and programs for controlling the operation of the shelf monitoring mobile robot; and The above-mentioned at least one processor is, At least one of the first imaging device and the second imaging device is controlled to photograph the first shelf facing the first side of the shelf monitoring mobile robot, and At least one of the third imaging device and the fourth imaging device is controlled to photograph the second shelf facing the second side of the shelf monitoring mobile robot, and A shelf monitoring system that generates real-gram data for at least one product provided on the first shelf and the second shelf based on an image of the first shelf and an image of the second shelf.

12. In Paragraph 11, The above-mentioned at least one processor is, Information regarding the position of the body at the time when the first shelf was photographed is obtained, and Information regarding the position of the body at the time when the second shelf was photographed is obtained, and Matching the image of the first shelf with information regarding the position of the body at the time the first shelf was photographed, and A shelf monitoring system that matches an image of the second shelf with information regarding the position of the body at the time the second shelf was photographed.

13. In Paragraph 12, The above-mentioned at least one processor is, The drive module is controlled so that the body moves to a first position closer to the first shelf than to the second shelf, and Controlling at least one of the third imaging device and the fourth imaging device to photograph the second shelf at the first position, and Controls the electronic shelf label identification device at the first position to acquire identification information of a first electronic shelf label adjacent to the first position provided on the first shelf, and Based on the identification information of the first electronic shelf label, information regarding the location of the first electronic shelf label is obtained, and A shelf monitoring system that matches an image of the second shelf taken at the first position with information regarding the position of the first electronic shelf label.

14. A shelf monitoring method for generating real-time real-gram data by photographing a first shelf provided on a first side of a shelf monitoring mobile robot and a second shelf provided on a second side of the shelf monitoring mobile robot different from the first side, A step of controlling at least one of a first imaging device and a second imaging device provided on the first side of the shelf monitoring mobile robot to photograph the first shelf; A step of controlling at least one of a third imaging device and a fourth imaging device provided on the second side of the shelf monitoring mobile robot to photograph the second shelf; and A shelf monitoring method comprising: a step of generating real-gram data for at least one product provided on the first shelf and the second shelf based on an image of the first shelf and an image of the second shelf.

15. In Paragraph 14, The step of controlling the first shelf to be photographed is: A step of controlling the first imaging device to photograph the first shelf; A step of detecting the upper edge and lower edge of the first shelf in an image of the first shelf; If the upper edge and lower edge of the first shelf are not detected in the image of the first shelf taken therefrom, the step of controlling the second imaging device to take an image of the upper region, central region, and lower region of the first shelf; and A shelf monitoring method comprising: a step of generating a first combined image by connecting images of the upper region, central region, and lower region of the first shelf.

Images