Shelf image generation system

Abstract

A shelf image generation system according to an embodiment includes: a plurality of cameras for capturing a shelf on which a plurality of products are displayed and a shelf image including a plurality of electronic price indicators corresponding to individual products; and an image server for generating a stitched shelf image by stitching a plurality of shelf images captured by the plurality of cameras, wherein the image server performs feature matching on the basis of an identification marker corresponding to a first electronic price indicator included in a first shelf image among the plurality of shelf images and an identification marker corresponding to a first electronic price indicator included in a second shelf image, and generates the stitched shelf image by connecting the first shelf image and the second shelf image according to a result of the feature matching.

Description

Shelf Image Generation System

[0001] Embodiments relate to a shelf image generation system, and more particularly, to a shelf image generation system for image stitching shelf images in a retail store captured by multiple cameras.

[0002] This refers to an e-paper terminal that displays real-time information, such as price and country of origin, for each product via a wireless network on store shelves. It's also called an Electronic Shelf Label (ESL). Utilizing an ESL offers the advantage of displaying desired information at any time, eliminating the need for frequent replacement of paper price tags. The market has recently been growing, centered around various retail outlets such as hypermarkets, supermarkets, health and beauty (H&B), and fashion stores.

[0003] The electronic shelf label system consists of a gateway, which handles wired and wireless communication, and tag devices that display product information. Using Internet of Things (IoT) technology, product information is transmitted from the ESL central management server to the gateway, which then wirelessly transmits it to multiple ESL tags. The tags display text, barcodes, and images.

[0004] Electronic paper (e-paper) displays, used in electronic shelf labels, are a technology that can replace the functions of existing displays and paper, such as liquid crystal displays (LCDs) and light-emitting diodes (LEDs). They consume less power and allow stored information to be viewed even when the power is off.

[0005] Electronic price tags can automatically and simultaneously reflect product-specific inventory status, price changes, and discount periods across all stores, improving store management efficiency. Stores can also reduce costs associated with printing price tags.

[0006] Various embodiments of the present disclosure provide a shelf image generation system capable of extracting unique points (key point extraction) for homography estimation using an electronic price tag and an identification code, thereby reducing the probability of failure and obtaining a successfully stitched shelf image.

[0007] The problems to be solved through the embodiments of the present disclosure are not limited to the problems described above, and problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the embodiments belong from this specification and the attached drawings.

[0008] A shelf image generation system according to one embodiment includes a plurality of cameras for capturing shelf images including a shelf on which a plurality of products are displayed and a plurality of electronic price tags corresponding to each product; and an image server for stitching a plurality of shelf images captured from each of the plurality of cameras to generate a stitched shelf image.

[0009] The above image server performs feature matching based on an identification marker corresponding to a first electronic price tag included in a first shelf image among the plurality of shelf images and an identification marker corresponding to the first electronic price tag included in a second shelf image, and connects the first shelf image and the second shelf image based on a result of the feature matching to generate the stitched shelf image.

[0010] The image server may calculate a homography matrix between the first shelf image and the second shelf image based on the result of the feature matching, and may generate the stitched shelf image after perspective transforming at least one image among the first shelf image and the second shelf image.

[0011] The above video server can assign different identification markers to each of the plurality of electronic price tags and control the display of the assigned identification markers on the corresponding electronic price tags.

[0012] The above video server can control the timing of displaying the identification marker and the shooting timing of the plurality of cameras to be synchronized.

[0013] The above video server can transmit price information to be displayed on the plurality of electronic price tags.

[0014] The above plurality of cameras are placed on a shelf facing the shelf on which the plurality of products are displayed, and calibration before shooting can be performed under the control of the image server.

[0015] The above image server can detect a plurality of electronic price tags included in the first shelf image, detect a plurality of electronic price tags included in the second shelf image, and recognize a pair of electronic price tags from a similarity calculation based on features extracted from identification markers of the plurality of electronic price tags detected in each shelf image.

[0016] The above image server can calculate a homography matrix based on the feature points of the recognized electronic price tag pair.

[0017] The above image server can generate inventory management information for the plurality of products based on the stitched shelf images.

[0018] The above identification marker may be an AruCo marker.

[0019] The shelf image generation system according to various embodiments of the present disclosure can extract unique points (key point extraction) for homography estimation by using an electronic price tag and an identification marker, thereby reducing the probability of failure and obtaining a successfully stitched shelf image compared to before.

[0020] The effects of the embodiments are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the embodiments belong from this specification and the attached drawings.

[0021] Figure 1a is a perspective view showing an electronic price indicator.

[0022] Figures 1b to 1d are exemplary drawings for explaining stitching shelf images according to the prior art.

[0023] FIG. 2 is a schematic diagram of a shelf image generation system according to one embodiment.

[0024] Figure 3 is a detailed schematic diagram of the image server (210) illustrated in Figure 2.

[0025] FIGS. 4A and 4B are exemplary drawings of an electronic price tag and an identification marker for shelf image stitching according to one embodiment.

[0026] FIGS. 5A and 5B are flowcharts illustrating a method for generating a shelf image according to another embodiment.

[0027] FIGS. 6A to 6F are exemplary drawings illustrating stitching of shelf images according to one embodiment.

[0028] The terms used in the examples are selected from widely used, current terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined not simply based on their names, but based on their meanings and the overall content of the present invention.

[0029] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "-unit" and "-module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0030] As used herein, when an expression such as "at least one" precedes an array of elements, it modifies the entire array of elements, not just each individual element. For example, the expression "at least one of a, b, and c" should be interpreted to include a, b, c, or a and b, a and c, b and c, or a and b and c.

[0031] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement them. The present disclosure may be implemented in a form that can be implemented in the aerosol generating devices of the various embodiments described above, or may be implemented in various different forms and is not limited to the embodiments described herein.

[0032] In the embodiment, an "electronic shelf label" or "electronic shelf label" is a device that displays specific information, such as a product name, price, and barcode, on a plastic tag. Also known as an Electric Shelf Label (ESL), ESL can be used. Gateway-related software can be installed where an electronic shelf label is needed, and the content to be displayed on the electronic shelf label can be set and applied. Here, the gateway acts as a public communication network between computers when multiple electronic shelf labels are interconnected with a network. For example, it acts as an intermediary between an electronic shelf label, a store POS device, and a product information management server. Through this intermediary, product information or product information is exchanged on the display of the electronic shelf label, such as electronic paper or an electronic panel, to display product information to consumers. Here, electronic paper (e-paper) is a display device that has a texture similar to regular paper, is easy on the eyes when viewed for a long time, and allows for free writing and erasing on the screen. It consumes little power, and the electronic ink (e-ink), the core of the e-paper, has a memory effect, allowing it to remain in place even when the power is cut off.

[0033] In the embodiment, the "user terminal" may be implemented as a computer or portable terminal that can access a server or other terminal via a network. Here, the computer may include, for example, a notebook, desktop, laptop, VR HMD (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.) equipped with a web browser. Here, the VR HMD includes all of the following: a PC-use (e.g., HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), a mobile-use (e.g., GearVR, DayDream, Storm Magic, Google Cardboard, etc.), a console-use (PSVR), and a stand-alone model (e.g., Deepon, PICO, etc.) that is implemented independently. A portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include not only a smart phone, a tablet PC, and a wearable device, but also various devices equipped with communication modules such as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, ultrasonic, infrared, WiFi, and LiFi. In addition, a "network" refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks.Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth, infrared, ultrasonic, visible light communication (VLC), and LiFi.

[0034] In an embodiment, image stitching means naturally or seamlessly joining photos of the same scene to create a single photo, and may be referred to as photo stitching or panoramic video.

[0035] In an embodiment, features can be divided into global features and local features. Global features represent the characteristics of an entire image or video. They represent the entire image and are information helpful in understanding the overall content of the image or video. Global features can encompass a large portion of the image or represent the overall distribution and statistical characteristics of the image. For example, in image classification tasks, global features can be used to determine the presence of objects or scenes by identifying the overall characteristics of the image. Global features can include the size of objects in the image, the overall color distribution, and texture. Local features, or regional features, represent the characteristics of small regions or sections of an image or video. These local features are used to represent the shape, color, texture, etc. of specific parts within the image. Local features are useful for identifying small patterns or structures within an image. In tasks such as object detection or image matching, local features can be utilized to recognize or match specific objects or patterns within the image. Local features are typically extracted from small image patches or window regions, and these features can be used to calculate or classify object or pattern similarities. In other words, global features represent the overall characteristics of an image and are used to understand its overall content, while local features represent the characteristics of small regions or parts and are used to analyze small patterns or structures. These features can be utilized in various tasks in image processing and computer vision.

[0036] In general, image stitching is possible when there is a common overlapping area between two images. This is achieved by calculating a homography matrix between the two images and applying a perspective transformation.

[0037] The homography matrix can be defined by the following mathematical equations 1 and 2.

[0038] [Mathematical Formula 1]

[0039]

[0040] [Equation 2]

[0041]

[0042] Here, X' is the transformed image, H is the homography matrix, and X is the actual image.

[0043] It has eight degrees of freedom and is estimated using at least four points pointing to the same point in the image, and is computed using the following steps.

[0044] 1. Keypoint detection and descriptor

[0045] Keypoint detection is the process of finding unique points in an image that can be easily distinguished from other points. Commonly used algorithms include, but are not limited to, Harris corner detection, Scale-Invariant Feature Transform (SIFT), Fast Algorithm for corner, and Super point.

[0046] Descriptors are extracted for keypoints for further matching processes during keypoint detection. Descriptors can be 64 or 128 floating-point values, depending on the descriptor type. Some descriptors may include, but are not limited to, Binary Robust Independent Elementary Features (BRIEF), Speeded Up Robust Features (SURF), and Superpoints.

[0047] 2. Feature Matching

[0048] Feature matching involves matching key points to find the ideal match for each point based on a distance metric. Some feature matching algorithms include, but are not limited to, K-Nearest Neighbors (KNN), Brute Force Matcher, Fast Library for Approximate Nearest Neighbors (FLANN), and Light Glue.

[0049] 3. Compute Homography

[0050] If there are matching points, a good match that gives a correct estimate can be used for estimation, as a homography matrix can be computed using RANSAC or minimum median to eliminate errors that may occur during the matching.

[0051] 4. Perspective Transforming

[0052] After obtaining the homography, the image must be transformed so that it can be converted to the plane of the reference image. To obtain the stitched image, the source image must be pasted in its original size.

[0053] Optionally, if you use different or multiple cameras to capture the images, you may need to calibrate the cameras and de-distort the images for image stitching.

[0054] Retail shelf images, as illustrated in Fig. 1b, are complex images with identical products arranged side by side. As can be seen in the images (101 and 102) illustrated in Fig. 1b, the same product is repeated in overlapping portions (111 and 112) of the images. The aforementioned key point matching, as illustrated in Fig. 1c, can be misrecognized because the globally matched points to the product may be incorrect due to the presence of identical products. As illustrated in Fig. 1d, mismatched key point features can lead to inappropriate estimation of homography with greater error and incorrect stitching.

[0055] In an embodiment, by using an electronic shelf label (ESL) for image stitching of a shelf in a retail store and an identification marker, such as an AruCo marker, displayed on the electronic shelf label, unique point extraction for homography estimation is possible, thereby reducing the probability of failure and obtaining a successfully stitched shelf image.

[0056] In this embodiment, a hybrid approach combining computer vision and deep learning is used to detect ESL devices, generate local features, and compute a similarity index using deep features to recognize matching ESL pairs. Later, the features calculated from the ESL markers are matched to estimate homography and perform image stitching.

[0057] In this embodiment, the entire shelf aisle can be inspected while reducing the number of cameras on the shelf to intelligently manage product inventory on the shelf. If image stitching is not performed due to overlapping camera areas, repeated products in the overlapping areas can cause problems in managing shelf inventory information. To improve existing image stitching results, the use of electronic shelf labels (ESLs) and identification markers, such as AruCo Markers, enables unique point extraction for homography estimation, reducing the failure rate and achieving successfully stitched shelf images.

[0058] FIG. 1a is a perspective view showing an electronic price indicator according to one embodiment.

[0059] Referring to FIG. 1, an electronic price tag (10) according to one embodiment may be placed or attached to a product display shelf in a distribution space to display product information (hereinafter, referred to as "product information"). For example, the electronic price tag (10) may receive product information from a server (not shown) and visually display the product information received from the server, thereby providing product information to a user. Depending on the embodiment, the electronic price tag (10) may also be referred to as an electronic shelf label (ESL) or an electronic information label (EIL).

[0060] According to one embodiment, the electronic price indicator (10) may include a housing (100) and a display (110).

[0061] The housing (100) can form the overall appearance of the electronic price indicator (10), and components of the electronic price indicator (10) can be arranged in the internal space of the housing (100). For example, a printed circuit board (not shown) including a processor for controlling the operation of the electronic price indicator (10), a communication module (not shown) for wireless communication with a server, and / or a battery (not shown) can be arranged in the internal space of the housing (100), but is not limited thereto.

[0062] According to one embodiment, the housing (100) may be formed in an overall rectangular parallelepiped shape as illustrated in FIG. 1, but the shape of the housing (100) is not limited to the illustrated embodiment. In another embodiment, the housing (100) may be formed in a polygonal columnar shape in addition to the rectangular parallelepiped shape.

[0063] The display (110) is positioned so that at least one area is exposed to the outer surface of the housing (100) and can output visual information. For example, the display (110) can output product information or product information received from a server, but is not limited thereto. In the present disclosure, product information may include at least one of price information, manufacturer, country of origin, discount information, or identification information (e.g., a barcode or QR code), and the expressions may be used with the same meaning below.

[0064] In an embodiment, the display (110) may output an identification marker to facilitate homography estimation for image stitching under the control of an ESL management server or an image server. Here, the identification marker may be an AruCo marker, but is not limited thereto, and may be a black-and-white matrix marker of a certain size that is easy to learn, etc. The AruCo marker will be described later with reference to FIG. 4D.

[0065] The display (110) can output product information through power supplied from a battery housed in the internal space of the housing (100).

[0066] FIG. 2 is a schematic diagram of a shelf image generation system according to one embodiment.

[0067] Referring to FIG. 2, the shelf image generation system includes a plurality of cameras (201 to 204) and a video server (210). Here, the plurality of cameras (201 to 204) may be positioned at facing shelf positions so as to be able to photograph the shelf, but is not limited thereto. The plurality of cameras (201 to 204) may capture shelf images including a shelf on which a plurality of products are displayed and a plurality of electronic price tags corresponding to each product. The range of the shelf that can be captured may vary depending on the specifications or angle of view of the camera. For example, when capturing a 10-meter-long shelf, three cameras positioned on opposite shelves spaced apart from each other may capture the shelf, and the individually captured shelf images may be connected to generate a 10-meter-long shelf image. Here, it goes without saying that the capture may be performed by three or more cameras.

[0068] The image server (210) stitches a plurality of shelf images captured from each of the plurality of cameras (201 to 204) to generate a stitched shelf image. The image server (210) performs feature matching based on an identification marker corresponding to a first electronic shelf label included in a first shelf image among the plurality of shelf images and an identification marker corresponding to a first electronic shelf label included in a second shelf image. The image server (210) connects the first shelf image and the second shelf image based on the result of the feature matching to generate a stitched shelf image. Specifically, the image server (210) may calculate a homography matrix between the first shelf image and the second shelf image based on the result of the feature matching, and may generate a stitched shelf image after perspective transforming at least one image among the first shelf image and the second shelf image. Here, feature matching, homography matrix calculation, perspective transform, etc. are as described with reference to the image stitching algorithm described above.

[0069] Here, the video server (210) can assign different identification markers to each of the plurality of electronic shelf labels and control the display of the assigned identification markers on the corresponding electronic shelf labels. At this time, the video server (210) can control the timing of displaying the identification markers and the shooting timing of the plurality of cameras (201 to 204) to be synchronized. Although not shown, an ESL server or shared gateway that controls the plurality of electronic shelf labels may be further included, and the video server (210) may be linked to an ESL server (not shown) or may function as a functional module of the ESL server. In this case, the video server (210) may also transmit price information to the electronic shelf labels.

[0070] Figure 3 is a detailed schematic diagram of the image server (210) illustrated in Figure 2.

[0071] Referring to FIG. 3, the video server (210) includes a camera control unit (211), an ESL control unit (212), a feature matching unit (213), a stitched image generation unit (214), and an inventory management information generation unit (215). In an embodiment, the video server (210) may perform, in addition to the function of stitching shelf images, a function of controlling multiple cameras, a function of controlling multiple electronic shelf labels (ESLs) arranged on a shelf, and a function of generating inventory management information for each product based on the stitched shelf images.

[0072] The camera control unit (211) controls multiple cameras that capture images of the shelf. The camera control unit (211) can control the cameras to begin capturing images in synchronization with the timing at which identification markers corresponding to electronic price tags placed on the shelf are displayed. The camera control unit (211) can control camera calibration to synchronize the multiple cameras.

[0073] The ESL control unit (212) generates identification markers to be displayed on electronic price tags placed on shelves. A unique identification marker can be generated for each of the multiple electronic price tags, and each electronic price tag or identification marker can be assigned a random ID for management. Here, the identification marker may be an Arco marker.

[0074] FIGS. 4A and 4B are exemplary drawings of an electronic price tag and an identification marker for shelf image stitching according to one embodiment.

[0075] Referring to Fig. 4a, an electronic price tag (401) may be placed at the bottom or at a specific location of products arranged on a shelf. Typically, product information, price information, etc. are displayed on the electronic price tag (401). In an embodiment, at a specific point in time, for example, when inventory management information must be generated, an identification marker (402) as shown in Fig. 4b is displayed on the electronic price tag. The identification marker may be displayed on the entire or a portion of the display area of ​​the electronic price tag (401). Here, the identification marker may be a reference marker or an Aruko marker.

[0076] An ARUKO marker is a synthetic rectangular marker consisting of a wide black border and an internal binary matrix that determines a unique identifier (ID). The black border facilitates rapid detection in an image, and the interior can be identified and applied to error detection and correction techniques through binary encoding. An ARUKO marker is one of the reference markers and consists of a two-dimensional bit pattern of size nxn and a black border area surrounding it. The marker size determines the size of the internal matrix. For example, a 4x4 marker size may consist of 16 bits. In an embodiment, each of a plurality of electronic shelf labels may be assigned a unique identification marker.

[0077] The feature matching unit (213) performs feature matching between identification markers existing in images in common areas or overlapping areas in shelf images captured by each camera.

[0078] The stitching image generation unit (214) calculates a homography matrix based on the matching points of each electronic price tag based on the feature matching results, and connects overlapping parts using the homography matrix and perspective transformation to generate a stitching image.

[0079] The inventory management information generation unit (215) can generate inventory information for products corresponding to IDs assigned to electronic price tags or identification markers based on the stitched images of the shelves. For example, the unit can determine the quantity of products from the product images corresponding to the electronic price tags present in the stitched shelf images. If the quantity of a product is insufficient, the unit can replenish the product or determine the availability of the product. Furthermore, the inventory management information generation unit (215) can also update the product database by linking with a separate product inventory management server (not shown).

[0080] In this embodiment, shelf image stitching can intelligently manage shelf inventory by reducing the number of cameras on a shelf while providing a full view of the entire aisle. Furthermore, it can address shelf inventory management issues caused by repeated products within the overlapping camera areas.

[0081] Figures 5a and 5b are flowcharts illustrating a method for generating shelf images according to another embodiment. In the embodiment, prior to generating shelf images or capturing shelf images, camera calibration of all cameras may be performed using Zhengyou Zhang's calibration method using a chess / checkerboard.

[0082] Referring to FIG. 5a, in step 500, an Arucomaker is generated. Here, the Arucomaker may be 4x4 or 4x5 in size, and based on this, an arbitrary ID may be generated for the electronic price tag or Arucomaker.

[0083] In step 502, a unique Aruko marker is assigned to the electronic shelf label. A unique Aruko marker is assigned to the electronic shelf label placed under the product on the shelf.

[0084] In step 504, the assigned Arucomaker is displayed on the electronic price tag.

[0085] In step 506, an image of the shelf including the Aruker marker is captured from a camera placed on the opposite shelf.

[0086] In step 508, image stitching is performed on shelf images containing overlapping regions or Arucomakers. Here, image stitching may include image undistorting, ESL detection, corresponding ESL pair detection, and image stitching through homography calculation, as illustrated in FIG. 5b. Here, camera calibration may be performed in step 512.

[0087] In step 510, shelf management is performed.

[0088] Referring to Fig. 6, the process of step 508 is described in detail.

[0089] First, once the camera calibration parameters (intrinsic and distortion) are estimated, the images captured from the cameras (601 and 602) are corrected / undistorted with distortion removed for each camera image, as illustrated in Fig. 6a. To accurately identify the locations of the electronic shelf labels, an object detection model specialized for the electronic shelf label dataset (yolov7-tiny, image size 1920) is used to detect the locations of the electronic shelf labels, and the electronic shelf labels are sorted by shelf row according to the shelf partition.

[0090] The identified electronic shelf labels shown in Fig. 6b are cut into rows, and a deep learning model (Efficient Net D3) specialized for the Aruko markers displayed on the target ESL is used to extract deep features, enhance contrast, calculate the cosine similarity of the corresponding images (611, 621) to identify matching images for the corresponding rows, and select the top three matches.

[0091] As illustrated in Fig. 6c, key points and corresponding features are extracted for each electronic price tag marker (621, 622). Here, SIFT features or Super Points may be used for feature extraction, but are not limited thereto.

[0092] As illustrated in Fig. 6d, the corresponding feature points of each electronic price tag (631, 632) are matched. Here, the K-Nearest and Lowes ratio tests or Light Glue matching can be used for feature matching.

[0093] As illustrated in Fig. 6e, a homography matrix is ​​calculated using the matched points of each electronic shelf label (641, 642).

[0094] As shown in Fig. 6f, a perspective transformation is applied to the image using a homography matrix and a perspective transformation to connect the overlapping portions.

[0095] Comparing the stitched shelf image according to the prior art (Fig. 1d) with the stitched shelf image according to another embodiment (Fig. 6f), it can be seen that the existing image stitching results are improved. That is, by using electronic price tags and identification markers, unique point extraction for homography estimation is possible, thereby reducing the failure rate compared to before and obtaining a successfully stitched shelf image.

[0096] The shelf image generation algorithm according to one embodiment may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, other data, such as program modules, in a modulated data signal, or other transport mechanism, and includes any information delivery media.

[0097] The description of the above-described embodiments is merely illustrative, and those skilled in the art will appreciate that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of protection for the invention should be defined by the appended claims, and all differences within the scope equivalent to the content described in the claims should be construed as being included within the scope of protection defined by the claims.

[0098] The present invention has industrial applicability in that it enables key point extraction for homography estimation by using an electronic price tag and an identification marker, thereby reducing the probability of failure compared to before and obtaining a successfully stitched shelf image, thereby increasing the efficiency of managing a store equipped with shelves.

Claims

1. A plurality of cameras for taking images of a shelf including a plurality of products displayed thereon and a plurality of electronic price tags corresponding to each product; and Including an image server that stitches multiple shelf images captured from each of the multiple cameras to create a stitched shelf image; The above video server, A shelf image generation system which performs feature matching based on an identification marker corresponding to a first electronic price tag included in a first shelf image among the plurality of shelf images and an identification marker corresponding to the first electronic price tag included in a second shelf image, and generates the stitched shelf image by connecting the first shelf image and the second shelf image according to a result of the feature matching.

2. In paragraph 1, The above video server, A shelf image generation system which calculates a homography matrix between the first shelf image and the second shelf image based on the result of the feature matching, and generates the stitched shelf image after perspective transforming at least one image among the first shelf image and the second shelf image.

3. In paragraph 1, The above video server, A shelf image generation system that assigns different identification markers to each of the plurality of electronic price tags and controls the display of the assigned identification markers on the corresponding electronic price tags.

4. In paragraph 3, The above video server, A shelf image generation system that controls the timing of displaying the above identification marker and the shooting timing of the plurality of cameras to be synchronized.

5. In paragraph 3, The above video server, A shelf image generation system that transmits price information to be displayed on the above plurality of electronic price tags.

6. In paragraph 3, The above multiple cameras, A shelf image generation system in which the above-mentioned plurality of products are placed on a shelf facing the shelf on which they are displayed, and calibration before shooting is performed under the control of the image server.

7. In paragraph 1, The above video server, Detecting a plurality of electronic price tags included in the first shelf image, Detecting multiple electronic price tags included in the second shelf image, A shelf image generation system that recognizes electronic shelf label pairs by calculating similarity based on features extracted from identification markers of multiple electronic shelf labels detected in each shelf image.

8. In paragraph 7, The above video server, A shelf image generation system that calculates a homography matrix based on feature points of the above-described recognized electronic shelf label pairs.

9. In paragraph 1, The above video server, A shelf image generation system that generates inventory management information for a plurality of products based on the stitched shelf images.

10. In paragraph 1, The above identification markers are, AruCo Marker, a shelf image generation system.

Images