Automatic labeling system for electronic shelf label

Abstract

This electronic shelf label labeling system forms a unique code of an electronic shelf label into a specific pattern that can be identified by a camera, so as to be identified by the camera, and groups the electronic shelf label location and displayed products on the basis of the identification so as to store and transmit data, and thus enables display management even in management of a large amount of products handled at large stores and in response to new products.

Description

Automatic labeling system for electronic shelf labels

[0001] The present invention relates to an electronic shelf labeling system capable of automatically performing labeling work that maps information on products to be displayed in a store.

[0002] Electronic shelf labels (ESL), which contain information about products displayed in offline stores, are widely used.

[0003] Meanwhile, in order to manage display through these electronic shelf labels, labeling work must be performed first to map information about the products to be displayed to each electronic shelf label.

[0004] However, electronic shelf labels can only receive information corresponding to their own unique codes via Wi-Fi, and it is impossible to manage where they are physically installed, so it is impossible to automatically match electronic shelf labels with displayed products, which is inconvenient for store managers to have to pair each electronic shelf label with products one by one.

[0005] The present invention provides an electronic shelf label labeling system that creates a unique code of an electronic shelf label in a store into a specific pattern identifiable by a camera, allows the camera to identify the code, and stores and transmits data by grouping the location of the electronic shelf label and the displayed products based on the identified code.

[0006] In addition, the present invention provides an electronic shelf labeling system that enables display management for large quantities of products handled in large stores and for responding to new products.

[0007] In addition, the present invention provides an electronic shelf label labeling system capable of automatically labeling product images using an electronic shelf label database (ESL DB) for products displayed in a store.

[0008] In addition, the present invention provides an electronic shelf label labeling system that generates and operates a bezel-based code for code identification in a low-resolution image.

[0009] In addition, the present invention provides an electronic shelf labeling system capable of responding to misdistribution without incurring unnecessary labor costs.

[0010] An electronic shelf labeling system according to one aspect of the disclosed invention comprises: a processor; an electronic shelf label (ESL) mounted on the front of a shelf configured to display products in rows by type set based on the front, the electronic shelf label (ESL) having a matrix code, which is a two-dimensional barcode capable of containing information on products displayed on the shelf; and a camera configured to photograph the front of the shelf to obtain an analysis target image including products displayed on the shelf and electronic shelf labels mounted at positions corresponding to the products, wherein the processor: extracts a product image and an electronic shelf label image included in the analysis target image through a first machine learning model based on the analysis target image; and controls a database memory in which information on the products is stored based on the electronic shelf label image so that shape information of the matrix code of the electronic shelf label is matched with information on the products corresponding to the product images.

[0011] In addition, the processor: determines whether to display the color of each cell constituting the matrix code based on the electronic shelf label image, thereby generating shape information of the matrix code; and controls a database memory in which product information is stored so that the shape information of the matrix code, in which the color display of each cell is determined, is matched with product information corresponding to the product image.

[0012] In addition, the electronic shelf label includes: a display unit configured to display a matrix code composed of cells arranged along a preset number of rows and a preset number of columns; and a bezel configured to surround the display unit, wherein the display unit can be configured to display an identifier area of ​​a color different from a color of the bezel at one end and the other end of the matrix code.

[0013] In addition, the processor may: determine the positions of each corner of the display unit in the electronic shelf label image based on the difference between the color of the identifier area included in the electronic shelf label image and the color of the bezel; extract a display unit image having each corner as a vertex from the electronic shelf label image based on the positions of each corner of the display unit; extract a binary code image, which is an image of an area excluding the identifier area, from the display unit image; and determine whether to display a color of each cell constituting the binary code image based on the binary code image, thereby generating shape information of the matrix code.

[0014] In addition, the display unit can display all cells of the column at one end and all cells of the column at the other end as identifier areas by displaying a color different from the color of the bezel.

[0015] In addition, the display unit: displays a matrix code composed of cells arranged along four rows and six columns; displays a color different from the color of the bezel as the identifier for all cells arranged in the first and sixth columns in the matrix code; and the processor: extracts images of cells arranged in the second, third, fourth, and fifth columns in the matrix code from an image of the display unit as the binary code image; and determines whether to display colors of cells arranged in the second, third, fourth, and fifth columns in the matrix code, thereby generating shape information of the matrix code.

[0016] In addition, the processor can generate an upscaled image by quadrupling the size of the electronic shelf label image to generate a high-resolution image through a second machine learning model based on a deep residual network structure based on the electronic shelf label image.

[0017] In addition, the processor: samples surrounding areas of each pixel of the up-scaled image based on the up-scaled image to measure the similarity between each pixel and the surrounding area; calculates a weight for each pixel based on the similarity between each pixel and the surrounding area; adjusts each pixel value of the up-scaled image based on the weight for each pixel to generate a noise-removed image, which is an image from which noise has been removed; and determines the positions of each corner of the display unit in the electronic shelf label image based on a difference between a color of the identifier area included in the noise-removed image and a color of the bezel.

[0018] In addition, the processor may: generate a light reflection-compensated binary code image based on the binary code image by a filtering method that normalizes light exposed in the binary code image; and generate shape information of the matrix code by determining whether to display the color of each cell constituting the binary code image based on the light reflection-compensated binary code image.

[0019] In addition, the processor: applies a logarithmic transform to the binary code image to generate a logarithmic transform image; Fourier transforms the logarithmic transform image to generate a Fourier transform image representing a frequency domain; separates a portion represented by a high-frequency domain from the Fourier transform image as an illumination element, and separates a portion represented by a low-frequency domain as a reflectance element; and generates a detail-enhancing image by adjusting the illumination element and the reflectance element in the Fourier transform image, wherein the illumination element represents a brightness variation of the image and a color tone variation due to a lighting condition as an intensity and distribution of illumination applied to the image, and the reflectance element can represent a characteristic of the image independent of a brightness variation of the image as color and detail information of the image.

[0020] In addition, the processor may: generate an inverse discrete Fourier transform image by performing an inverse discrete Fourier transform on the detail highlight image; and generate the light reflection compensation binary code image by applying an exponential transform to the inverse discrete Fourier transform image.

[0021] In addition, the processor: generates a marker restoration image by performing binarization by merging a center surround contour that places weight on the center of the image while increasing a threshold for brightness intensity based on the light reflection compensation binary code image; and generates shape information of the matrix code by determining whether to display a color of each cell constituting the binary code image based on the marker restoration image.

[0022] In addition, the system may further include a machine learning unit configured to learn the first machine learning model through a machine learning method by photographing the front of the learning shelf, setting a learning image including learning products displayed on the learning shelf and learning electronic shelf labels mounted at positions corresponding to the learning products as input variables, and setting an area where the learning products are located and an area where the learning electronic shelf labels are located in the learning image as output variables.

[0023] In addition, the processor can control a database memory in which product information is stored so that, based on the electronic shelf label image, shape information of the matrix code of the electronic shelf label matches at least one of product type information, product price information, product arrival time information, and product weight information corresponding to the product image.

[0024] In addition, the processor can control the database memory to classify and store information on a product corresponding to the product image and shape information on a matrix code for which matching is completed as a matching completed matrix code.

[0025] In addition, the camera is configured to capture the front of the inspection target shelf to determine whether the displayed products are misdisplayed, and to obtain an inspection target image including the inspection target products displayed on the inspection target shelf and an electronic shelf label mounted at a position corresponding to the inspection target products, and the processor can determine whether shape information of a matrix code generated based on the inspection target image is identical to a matching completed matrix code corresponding to the inspection target products.

[0026] In addition, the system communication unit is further configured to be able to communicate with a terminal of a manager who manages the shelf, and the processor: if the shape information of the matrix code generated based on the inspection target image is different from the matching completed matrix code corresponding to the inspection target product, generates mis-display information to the effect that the product display should be re-arranged for the inspection target shelf; and controls the system communication unit to transmit the mis-display information to the manager's terminal.

[0027] In addition, the electronic shelf label includes a label communication unit configured to receive a signal generated by the processor, and the processor: generates a code change signal, which is a signal for controlling the electronic shelf label of the inspection target shelf so that the electronic shelf label of the inspection target shelf displays the matching completion matrix code, when the shape information of the matrix code generated based on the inspection target image is different from the matching completion matrix code corresponding to the inspection target product; and transmits the code change signal to the electronic shelf label of the inspection target shelf.

[0028] According to one aspect of the disclosed invention, a control method for an electronic shelf labeling system comprising an electronic shelf label mounted on the front of a shelf configured to allow products to be displayed along preset rows by type based on the front, and a matrix code, which is a two-dimensional barcode capable of containing information on products displayed on the shelf, a camera, and a processor, may include the steps of: photographing the front of the shelf by the camera to obtain an analysis target image including products displayed on the shelf and electronic shelf labels mounted at positions corresponding to the products; extracting, by the processor, a product image and an electronic shelf label image included in the analysis target image through a first machine learning model based on the analysis target image; determining, by the processor, whether to display a color of each cell constituting the matrix code based on the electronic shelf label image to generate shape information of the matrix code; and controlling, by the processor, a database memory in which information on the products is stored so that the shape information of the matrix code, in which whether to display a color of each cell is determined for each cell, is matched with information on the products corresponding to the product images.

[0029] A non-transitory recording medium according to one aspect of the disclosed invention can store a computer-readable computer program to execute a control method of an electronic shelf label labeling system.

[0030] According to one aspect of the disclosed invention, a unique code of an electronic shelf label in a store can be made into a specific pattern identifiable by a camera, and the camera can identify the unique code, and based on this, the location of the electronic shelf label and the displayed products can be linked to store and transmit data.

[0031] In addition, according to an embodiment of the present invention, display management can be made possible for large-scale product management and new product response in large stores.

[0032] In addition, according to an embodiment of the present invention, automatic labeling of product images using an electronic shelf label database (ESL DB) for products displayed in a store may be possible.

[0033] Additionally, according to an embodiment of the present invention, a bezel-based code for code identification in a low-resolution image can be generated and operated.

[0034] In addition, according to an embodiment of the present invention, it is possible to respond to misprinting without incurring unnecessary labor costs.

[0035] FIG. 1 is a control block diagram of an electronic shelf label labeling system according to one embodiment.

[0036] Figure 2 is a drawing for explaining how a product image is extracted from an image to be analyzed.

[0037] Figure 3 is a drawing for explaining how an electronic shelf label image is extracted from an analysis target image.

[0038] FIG. 4 is a drawing illustrating an electronic shelf label image extracted according to one embodiment.

[0039] FIG. 5 is a drawing illustrating an electronic shelf label image including a display unit displaying an identifier area according to one embodiment.

[0040] FIG. 6 is a diagram illustrating extraction of a binary code image from an electronic shelf label image according to one embodiment.

[0041] FIG. 7 is a diagram illustrating a process for generating a light reflection compensation binary code image according to one embodiment.

[0042] FIG. 8 is a diagram illustrating generating a marker restoration image according to one embodiment.

[0043] Fig. 9 is a flowchart of a control method of an electronic shelf label labeling system according to one embodiment.

[0044] Throughout the specification, the same reference numerals denote the same components. This specification does not describe all elements of the embodiments, and any content that is general in the technical field to which the disclosed invention belongs or that overlaps between the embodiments is omitted. The term "~unit" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "~units" may be implemented as a single component, or a single "~unit" may include multiple components.

[0045] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0046] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0047] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0048] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0049] The operating principle and embodiments of the disclosed invention are described below with reference to the attached drawings.

[0050] FIG. 1 is a control block diagram of an electronic shelf label labeling system according to one embodiment.

[0051] Referring to FIG. 1, an electronic shelf label labeling system (100) according to an embodiment of the present invention may include a processor (110), an electronic shelf label (120), a camera (130), a database memory (140), a machine learning unit (150), a system communication unit (160), and a memory (170).

[0052] The electronic shelf labeling system (100) may be utilized by a store that sells products displayed on shelves. The electronic shelf labeling system (100) may be installed on a terminal within the store or on a server within the store. Alternatively, the store may utilize the electronic shelf labeling system (100) by having the terminal communicate with the electronic shelf labeling system (100) located on a central server. The store may be an unmanned store without employees or a staffed store.

[0053] Store shelves can be arranged in preset rows, with products categorized by type, with the front facing the front. The displayed products can be arranged in rows so that their names are clearly visible.

[0054] Electronic shelf labels (ESL) can be mounted on the front of a shelf. The ESL can display a matrix code (300), a two-dimensional barcode that can contain information about products displayed on the shelf. Generally, if product display is performed normally, information about products displayed on the top of the shelf where an ESL is located can be contained in the matrix code (300) of the ESL. Store employees, store managers, or customers can scan the matrix code (300) of the ESL with their terminals and check the product information contained in the matrix code (300) through the terminals.

[0055] Electronic display labels that display matrix codes (300) containing product information in this way can easily convey information about products displayed in locations corresponding to the electronic display labels to users. However, the process of pairing these matrix codes (300) with each product is still performed by store managers or employees. This process is excessively time-consuming and can lead to the matrix codes (300) being incorrectly paired with the wrong products, resulting in the matrix codes (300) of the electronic display labels mounted on the bottom of certain products containing information about completely different products. Therefore, a system is needed that automatically pairs the matrix codes (300) of each electronic display label with each product, rather than requiring human intervention.

[0056] The camera (130) can capture the front of the shelf to obtain an analysis target image (200). The analysis target image (200) may be an image including products displayed on the shelf and electronic shelf labels (120) mounted at locations corresponding to the products. That is, for example, in a store display environment, a camera (130) can be installed on a shelf displaying products and monitor the display status of products placed on the opposite shelf by capturing the image. The camera (130) can be positioned so that the area of ​​the image it captures overlaps with that of an adjacent camera (130) by about 1 / 3, depending on the distance from the shelf. The cameras (130) can be sequentially positioned along the store aisles and each can be managed by assigning an index. The camera (130) can transmit the analysis target image (200) to the processor (110).

[0057] Figure 2 is a drawing for explaining how a product image is extracted from an analysis target image, and Figure 3 is a drawing for explaining how an electronic shelf label image is extracted from an analysis target image.

[0058] Referring to FIGS. 1, 2, and 3, the processor (110) can extract a product image (210) and an electronic shelf label image (220) included in the analysis target image (200) through a first machine learning model (171) based on the analysis target image (200).

[0059] Object detection in images can be done using traditional methods or methods utilizing deep neural networks (DNNs). Traditional methods, while in various forms, commonly involve pre- and post-processing of the image, feature extraction for classification, and finally, object region detection using these features.

[0060] Meanwhile, the processor (110) extracting the electronic shelf label image (220) through the first machine learning model (171) based on the analysis target image (200) may detect the electronic shelf label (ESL) (120) attached to the shelf by utilizing the DNN (Deep Neural Network) included in the first machine learning model (171). At this time, the processor (110) can estimate the area of ​​the actual object through the bounding box of the area similar to the shape of the object learned as a method of distinguishing the background using the first machine learning model (171) and the probability information that the box is the object. Representative models include R-CNN, YOLO, ViT, etc. However, the type of the first machine learning model (171) is not limited thereto.

[0061] Meanwhile, since the electronic shelf labels (120) are attached to the rails in front of the shelves with uniform sizes, there may be cases where the rails, which are shelf configurations between one electronic shelf label (120) and another electronic shelf label (120), are identified as electronic shelf labels (120) by light reflection. This may not be removed in the step of extracting the image of the electronic shelf label (120), but may be classified as an outlier in the subsequent step of identifying the unique code within the tag of the electronic shelf label (120).

[0062] The first machine learning model (171) can be stored in a memory (170) provided in the electronic shelf label labeling system (100).

[0063] The learning image may be an image of the front of a learning shelf, including learning products displayed on the learning shelf and learning electronic shelf labels (120) mounted at positions corresponding to the learning products. The learning shelf may be a shelf in which information on each learning product already accurately displayed is contained in a matrix code (300) of an electronic shelf label (120) mounted at positions corresponding to each learning product, in order to train the first machine learning model (171).

[0064] The machine learning unit (150) can learn the first machine learning model (171) through a machine learning method by setting the learning image as an input variable and setting the area where the learning product is located and the area where the learning electronic shelf label is located in the learning image as output variables.

[0065] The machine learning unit (150) can learn the first machine learning model (171) by using the outputs, which are the final results (outputs) produced by the first machine learning model (171) according to the input. The machine learning unit (150) may be provided in the electronic shelf label labeling system (100), but is not necessarily limited thereto. For example, the electronic shelf label labeling system (100) may receive the first machine learning model (171) generated by the machine learning unit (150) provided in another device or server from the device or server and store the first machine learning model (171) in the memory (170).

[0066] Machine learning utilizes models composed of multiple parameters and can mean optimizing those parameters based on given data. Depending on the type of learning problem, machine learning can include supervised learning, unsupervised learning, and reinforcement learning. Supervised learning learns mappings between inputs and outputs and is applicable when input-output pairs are given as data. Unsupervised learning is applicable when there are only inputs and no outputs, and can identify patterns between inputs, etc.

[0067] FIG. 4 is a drawing illustrating an electronic shelf label image extracted according to one embodiment.

[0068] Referring to FIG. 4, the electronic shelf label (120) may include a display unit (121) and a bezel (122). The display unit (121) may display a matrix code (300) composed of cells arranged along a preset number of rows and a preset number of columns. The display unit (121) may simply be paper or plastic on which the matrix code (300) is printed, but is not limited thereto. For example, the display unit (121) may be a display including a liquid crystal display. In this case, the display unit (121) of any one electronic shelf label (120) may not fix the matrix code (300) it displays to a initially set value, but may change it differently depending on the display.

[0069] The bezel (122) may be configured to surround the display unit (121). For example, if the display unit (121) is a display, the bezel (122) may be a plastic or metal material configuration surrounding the display. The color of the bezel (122) may be white, but is not limited thereto, and any color that is distinct from the color displayed in the matrix code (300) may be the color of the bezel (122).

[0070] Referring to FIGS. 1 and 4, the processor (110) can control the database memory (140) based on the electronic shelf label image (220).

[0071] The database memory (140) may be a store database where product information is stored. Product information may include product type information, product price information, product arrival time information, product weight information, product quantity or inventory information, etc.

[0072] The processor (110) can control the database memory (140) in which product information is stored so that the shape information of the matrix code (300) of the electronic shelf label (120) is matched with the product information corresponding to the product image (210) based on the electronic shelf label image (220). At this time, the product image (210) itself as the product information may be matched with the shape information of the matrix code (300) of the electronic shelf label (120), or the name of the product derived through the product image (210) may be matched with the shape information of the matrix code (300) of the electronic shelf label (120). In addition, among the numerous product images (210) included in the analysis target image (200), the product information matching the shape information of the matrix code (300) of one electronic shelf label image (220) may be the product information corresponding to the product image (210) located closest to the corresponding electronic shelf image.

[0073] The processor (110) can determine whether each cell constituting the matrix code (300) is colored based on the electronic shelf label image (220), thereby generating shape information of the matrix code (300). That is, the processor (110) can determine whether each cell of the matrix code (300) included in the electronic shelf label image (220) is a colored cell or a non-colored cell, thereby determining what kind of two-dimensional barcode shape the overall shape of the matrix code (300) has. The shape information of the matrix code (300) generated by the processor (110) can be an estimated two-dimensional barcode shape of the matrix code (300) displayed by the electronic shelf label (120) of the corresponding electronic shelf label image (220), based on the electronic shelf label image (220) included in the analysis target image (200).

[0074] The processor (110) can control the database memory (140) in which product information is stored so that the shape information of the matrix code (300) in which the color display of each cell is determined is matched with the product information corresponding to the product image (210).

[0075] Specifically, the processor (110) can control the database memory (140) in which product information is stored so that the shape information of the matrix code (300) of the electronic shelf label (120) is matched with at least one of the product type information, product price information, product arrival time information, and product weight information corresponding to the product image (210) based on the electronic shelf label image (220).

[0076] Meanwhile, the electronic shelf label labeling system (100) can determine whether products are displayed in their original planned locations for each shelf.

[0077] The processor (110) can control the database memory (140) to classify and store the shape information of the matrix code (300) that has been matched with the product information corresponding to the product image (210) as the matching completed matrix code (300).

[0078] The shelf under inspection may be a shelf in a store where it is desired to determine whether the displayed products are misdisplayed. The product under inspection may be the product displayed on the shelf under inspection and subject to inspection for misdisplay.

[0079] The camera (130) can capture the front of the inspection target shelf to obtain an inspection target image. The inspection target image may include an image of the inspection target product displayed on the inspection target shelf and an electronic shelf label (120) mounted at a location corresponding to the inspection target product. The camera (130) can transmit the inspection target image to the processor (110).

[0080] The processor (110) can determine whether the shape information of the matrix code (300) generated based on the inspection target image is identical to the matching completed matrix code (300) corresponding to the inspection target product.

[0081] If the shape information of the matrix code (300) generated based on the inspection target image is different from the matching completed matrix code (300) corresponding to the inspection target product, the processor (110) may generate misdisplay information. The misdisplay information may be information or a signal to re-arrange the product display on the inspection target shelf.

[0082] The system communication unit (160) may be configured to communicate with a terminal (400) of a manager who manages shelves. The processor (110) may control the system communication unit (160) to transmit misalignment information to the terminal (400) of the manager.

[0083] When a misdisplay is detected for a product displayed on a shelf, the processor (110) can update information on each abnormal situation in the database memory (140).

[0084] The processor (110) can generate an abnormal situation notification. The processor (110) can accumulate and register the part about the out-of-stock and mis-display events in the database memory (140) and generate a notification to be delivered to the administrator. The processor (110) can initialize or leave information about the abnormal situation in the database memory (140) according to the administrator's interaction (user-interaction) with the corresponding notification. When the administrator completes the response to the abnormal situation, the database memory (140) about the abnormal situation can be initialized. If the administrator has not yet completed the response to the abnormal situation, the processor (110) can accumulate information about the new abnormal situation to the existing abnormal situation information when an additional abnormal situation occurs and update the database memory (140) so that the administrator can check and respond again at any time.

[0085] Meanwhile, if the information displayed on each product and the corresponding electronic shelf label (120) is different, incorrectly displayed products may be moved as described above, but the display of the electronic shelf label (120) itself may also be changed to match the information on the corresponding products.

[0086] For example, when the display locations of products in a store change due to reasons such as a misdisplay of products or the arrival of new products, the display of each electronic shelf label (120) may be automatically changed so that information corresponding to each product is displayed without the manager having to individually change the display of each electronic shelf label (120). In this case, the electronic shelf label labeling system (100) may control the display unit (121) of the electronic shelf label (120) so that the electronic shelf label (120) indicating each product displays a matrix code (300) corresponding to the products for each shelf.

[0087] The electronic shelf label (120) may include a label communication unit (123). The label communication unit (123) may receive a signal generated by the processor (110).

[0088] If the shape information of the matrix code (300) generated based on the inspection target image is different from the matching completion matrix code (300) corresponding to the inspection target product, the processor (110) may generate a code change signal. The code change signal may be a signal that controls the electronic shelf label (120) of the inspection target shelf so that the electronic shelf label (120) of the inspection target shelf displays the matching completion matrix code (300).

[0089] The processor (110) can transmit a code change signal to the electronic shelf label (120) of the shelf to be inspected. At this time, the electronic shelf label (120) that has received the code change signal can display a matching completion matrix code (300).

[0090] Meanwhile, in order to identify which electronic shelf label (ESL) (120) detected on the analysis target image (200) is in the ESL management DB, a unique code capable of expressing each electronic shelf label is required. Due to limitations in the liquid crystal size and shooting distance of the display unit (121) of the electronic shelf label (120) that will display the code, the identifier for identifying the marker must be simple and not occupy a narrow space in the display. In addition, due to the various specifications of the electronic shelf label (120), there may not be enough space for the identifier.

[0091] The electronic shelf label labeling system (100) generates a 4x4 sized binary grid code that fills the narrow axis of the display to solve the above-mentioned problem, and fills the remaining space for the wide axis with a color that contrasts with the color of the bezel (122), thereby using the corners generated from the contrast between the bezel (122) and the marker margin as identifiers, and extracts a binary code image (222) by cutting out an area corresponding to the width of the narrow axis from the center of the four corners.

[0092] FIG. 5 is a drawing illustrating an electronic shelf label image including a display unit displaying an identifier area according to one embodiment.

[0093] Referring to FIGS. 4 and 5, the display unit (121) can display an identifier area (301). The identifier area (301) is located at one end and the other end of the matrix code (300), and may be an area of ​​a different color from the color of the bezel (122).

[0094] Specifically, the display unit (121) can display all cells of a column at one end and all cells of a column at the other end as identifier areas (301) by displaying a color different from the color of the bezel (122) in all cells of a column at one end and in all cells of a column at the other end. For example, the display unit (121) can display the color of all cells of a column at the left end and in all cells of a column at the right end as black, thereby displaying them differently from the color of the bezel (122), which is white.

[0095] FIG. 6 is a diagram illustrating extraction of a binary code image from an electronic shelf label image according to one embodiment.

[0096] Referring to FIGS. 1, 5, and 6, the processor (110) can determine the positions of each corner of the display portion (121) in the electronic shelf label image (220) based on the difference between the color of the identifier area (301) included in the electronic shelf label image (220) and the color of the bezel (122).

[0097] The processor (110) can extract a display image (221) based on the positions of each corner of the display unit (121). The display image (221) may be an image with each corner as a vertex in the electronic shelf label image (220). However, the vertices of the display image (221) do not necessarily have to be located at positions that exactly correspond to the positions of each corner. For example, the vertices of the display image (221) may be located near each corner.

[0098] The processor (110) can extract a binary code image (222) from the display image (221). The binary code image (222) may be an image of an area excluding the identifier area (301) in the display image (221).

[0099] The processor (110) can generate shape information of the matrix code (300) by determining whether to display the color of each cell constituting the binary code image (222) based on the binary code image (222).

[0100] Meanwhile, considering the resolution or number of pixels of the electronic shelf label image (220), it is desirable to display the matrix code (300) with the number of rows and columns that do not break the image of each cell of the binary code image (222) while including as much information as possible.

[0101] The display unit (121) can display a matrix code (300) composed of cells arranged along four rows and six columns. That is, the matrix code (300) can be a 6x4 two-dimensional barcode.

[0102] The display unit (121) can display a color different from the color of the bezel (122) as an identifier in all cells arranged in the first and sixth columns of the matrix code (300). That is, the part that actually contains product information, excluding the identifier, can be a 4x4 two-dimensional barcode.

[0103] The processor (110) can extract images of cells arranged in the 2nd, 3rd, 4th, and 5th columns of the matrix code (300) from the image of the display unit (121) as binary code images (222).

[0104] The processor (110) can generate shape information of the matrix code (300) by determining whether to display the colors of the cells arranged in the 2nd, 3rd, 4th, and 5th columns in the matrix code (300).

[0105] Meanwhile, the analysis target image (200) captured by the camera (130) may be an image including a large shelf, numerous products displayed on the shelf, and electronic shelf labels (120) corresponding to each product. At this time, the area occupied by each electronic shelf label (120) image in the analysis target image (200) may be relatively very small. In other words, the electronic shelf label image (220) may be a low-resolution image with an excessively small total pixel count. The electronic shelf label labeling system (100) needs to accurately determine the shape of the matrix code (300) actually displayed by the electronic shelf label (120) of the electronic shelf label image (220) based on such low-resolution electronic shelf label image (220).

[0106] Image restoration is essential for pattern recognition within ultra-low-resolution images. Common image restoration techniques include denoising using Gaussian or nonlocal means, upscaling (e.g., Super-resolution) to restore image loss and enlarge the image, and other methods to remove shadows or restore hidden areas. Finding the bezel (122) and the corner boundaries of the code requires prior image upscaling and denoising. Because the display representing the actual code is less than 20 pixels, even a pixel or two of error can significantly impact area calculations.

[0107] An electronic shelf label labeling system (100) according to one embodiment can perform upscaling using the commonly used EDSR x4 super resolution technique and minimize noise using a noise removal method (NonLocal Means Denoising) that organizes pixels based on the similarity between adjacent pixels.

[0108] The EDSR x4 super-resolution technique is one of the image super-resolution methods that utilizes deep learning. EDSR (Enhanced Deep Super-Resolution) can transform low-resolution images into high-resolution images to improve their resolution. EDSR can be performed based on a deep residual network (DRN) structure. DRNs can mitigate the gradient vanishing problem that can occur in deep networks by utilizing skip connections between layers. This allows EDSR to perform more sophisticated image transformations using a deeper network. EDSR is particularly effective at x4 super-resolution, which quadruples the size of a low-resolution image to create a high-resolution image. EDSR can restore the details and edges of high-resolution images, producing sharper and more natural-looking images.

[0109] The noise removal method performed by the electronic shelf labeling system (100) according to one embodiment may be a method using the Non-Local Means Denoising technique. Non-Local Means Denoising may be one of the algorithms for removing image noise. This algorithm may be a method for generating a cleaner image by removing noise by utilizing the texture and pattern of the image. The Non-Local Means Denoising algorithm may be composed of a similarity measurement step, a weight calculation step, and a noise removal step in that order.

[0110] The similarity measurement step may involve sampling the surrounding area of ​​each pixel in the input image and measuring the similarity between that pixel and the surrounding area. This similarity may be an indicator of how similar the textures or patterns in the surrounding area are.

[0111] The weight calculation step can calculate a weight for each pixel based on similarity. A higher weight can have a greater impact on a pixel as the similarity increases. This weight can be used to estimate the value of a pixel by leveraging information from other pixels in the input image.

[0112] The noise removal step can remove noise by adjusting the value of each pixel in the input image using weights. Pixels with large weights can estimate more accurate values ​​by leveraging information from surrounding pixels.

[0113] The aforementioned NonLocal Means Denoising technique utilizes image textures and patterns to remove noise, making it effective in creating more natural and detailed images. Furthermore, because it utilizes information from surrounding areas to remove noise, it can remove noise while taking global features into account.

[0114] The processor (110) can generate an upscaling image by quadrupling the size of the electronic shelf label image (220) to generate a high-resolution image through a second machine learning model (172) based on the electronic shelf label image (220). At this time, the processor (110) can generate the upscaling image using the EDSR x4 super resolution technique.

[0115] The second machine learning model (172) may be a machine learning model based on a deep residual network structure. The second machine learning model (172) may be stored in a memory (170) provided in the electronic shelf label labeling system (100).

[0116] The processor (110) can measure the similarity between each pixel and the surrounding area by sampling the surrounding area of ​​each pixel of the upscaled image based on the upscaled image.

[0117] The processor (110) can calculate a weight for each pixel based on the similarity between each pixel and the surrounding area.

[0118] The processor (110) can generate a noise-removed image, which is an image with noise removed, by adjusting each pixel value of the upscaled image based on a weight for each pixel.

[0119] The processor (110) can determine the location of each corner of the display unit (121) in the electronic shelf label image (220) based on the difference between the color of the identifier area (301) included in the noise removal image and the color of the bezel (122).

[0120] Meanwhile, even if the display image (221) is extracted based on the positions of each corner determined in the above-described manner and the binary code image (222) is extracted to complete the preprocessing process, the surface of the electronic shelf label (120) in the electronic shelf label image (220) may have a mixture of lighting and reflection components, and a non-uniform phenomenon of features due to the reflection angle may remain. This may cause an error when generating shape information of the matrix code (300) of the electronic shelf label image (220), so it may be desirable to generate an image in which light reflection is corrected for the binary code image (222), and to generate shape information of the matrix code (300) based on the image in which light reflection is corrected.

[0121] FIG. 7 is a diagram illustrating a process for generating a light reflection compensation binary code image according to one embodiment.

[0122] Referring to FIG. 7, the processor (110) can perform boundary-based binarization in the frequency domain using a homomorphic filter technique. The homomorphic filter technique is a technique mainly used when normalizing light exposed to an image, and can be expressed as a product of an illumination element and a reflectance element. The illumination element can represent the light exposed to the image (low frequency), and the reflectance element can represent the edge of objects existing in the image (high frequency). In order to process this into an image, a logarithmic operation is taken to change the multiplication operation into a sum operation, and by removing the logarithm through a high pass filter and an exponential (exp) operation, an image with the illumination removed can be obtained.

[0123] Illumination elements can represent the lighting conditions of an image. This can refer to the intensity and distribution of the light applied to the image. Illumination elements can include variations in the image's brightness or changes in color tone due to lighting conditions. In the homomorphic filter technique, the portion representing the high-frequency region in the frequency domain after log transformation can be interpreted and adjusted as the illumination element.

[0124] Reflectance elements can refer to the reflective properties of real-world objects. They can represent the color, texture, and details of objects in an image. Reflectance elements interact with lighting elements to preserve the characteristics of real-world objects independently of brightness fluctuations in the image. In the homomorphic filter technique, the portion representing the low-frequency region in the frequency domain after log transformation can be interpreted and extracted as a reflectance element.

[0125] The goal of this homomorphic filter technique may be to separate the illuminating and reflective elements, manipulate them individually, and then recombine them to produce a more balanced image. That is, the processor (110) can generate an image with the illumination removed, which compensates for brightness fluctuations in the image by adjusting the illuminating elements and enhances details by emphasizing the reflective elements.

[0126] The processor (110) can restore a marker to obtain an image robust to light reflection by applying an image with the lighting removed to a gray image, increasing a weight (threshold) for brightness intensity for the restored image, merging a center surround contour (a contour extraction method that places weight on the center of the image), and performing binarization. Thereafter, the processor (110) can classify a cluster occupying a majority of pixels for each cell in a 4x4 grid as a representative pattern of the corresponding cell and identify a code.

[0127] As mentioned above, the homomorphic filter technique is an image processing technique used to normalize the light exposure to an image. This method can be primarily used to extract sharp information from images under different lighting conditions or to control the dynamic range of an image. The homomorphic filter technique primarily operates in the frequency domain. This method can isolate and adjust the brightness and detail of an image by combining logarithmic transformation and filtering operations.

[0128] Such a homomorphic filter technique may include a step of generating a light reflection-corrected binary code image (g(x,y)) by performing a logarithmic transform (ln) on the binary code image (f(x,y))(222), a Fourier transform (DFT) step, a filtering step (H(u,v)), an inverse transform step (DFT^(-1)), and an exponential transform (exp).

[0129] The log transformation step (ln) may be a step that applies a log transformation to the input image, a binary code image (f(x,y))(222). The log transformation can serve to expand the dynamic range of the image, reducing the difference between bright and dark areas. This can mitigate the effect of brightness that varies depending on lighting conditions.

[0130] The Fourier transform (DFT) step, which converts the log-transformed image to the frequency domain, can be a step that converts the log-transformed image to the frequency domain. For this purpose, the discrete Fourier transform can be primarily used.

[0131] The filtering step (H(u,v)) may be a step that performs a filtering operation in the frequency domain. At this time, low-pass filtering and high-pass filtering are often combined to compensate for brightness changes in the image and emphasize details.

[0132] The inverse transform step (DFT^(-1)) may be a step to inversely transform the filtered frequency domain image back to the spatial domain. For this, the inverse discrete Fourier transform (IDFT) can be used.

[0133] The exponential transform step (exp) may be a step of applying an exponential transform to the inversely transformed image to restore the original image dynamic range and generate a reflection-compensated binary code image (g(x,y)).

[0134] Specifically, referring to FIG. 1 and FIG. 7, the processor (110) can generate a light reflection compensation binary code image based on the binary code image (222) by a filtering method that normalizes the light exposed in the binary code image (222).

[0135] The processor (110) can generate shape information of the matrix code (300) by determining whether to display the color of each cell constituting the binary code image (222) based on the light reflection compensation binary code image.

[0136] The processor (110) can apply a log transformation to a binary code image (222) to generate a log transformation image.

[0137] The processor (110) can generate a Fourier transform image representing a frequency domain by Fourier transforming a log transform image.

[0138] The processor (110) can separate a portion expressed in a high-frequency region from a Fourier transform image as an illumination element, and separate a portion expressed in a low-frequency region as a reflectance element.

[0139] The processor (110) can generate a detail-enhanced image by adjusting the lighting element and the reflection element in the Fourier transform image.

[0140] At this time, the lighting element represents the intensity and distribution of lighting applied to the image, which indicates changes in brightness of the image and changes in color tone due to lighting conditions, and the reflection element represents the characteristics of the image that are independent of changes in brightness of the image, as color and detail information of the image.

[0141] The processor (110) can generate an inverse discrete Fourier transform image by performing an inverse discrete Fourier transform on the detail-enhanced image.

[0142] The processor (110) can generate a light reflection compensation binary code image by applying an exponential transform to an inverse discrete Fourier transform image.

[0143] FIG. 8 is a diagram illustrating generating a marker restoration image according to one embodiment.

[0144] Referring to FIG. 8, the electronic shelf label labeling system (100) can restore markers to be robust to light reflection by performing binarization by merging center surround contours while increasing the threshold for brightness intensity. The aforementioned restoration method can be utilized as a preprocessing step to recover details lost due to light reflection in an image and to minimize the effects of light reflection.

[0145] The method for restoring markers can be composed of a threshold setting step, a binarization step, a center-surround contour merging step, and a marker restoration step.

[0146] The threshold setting step may be a step that sets a certain threshold to minimize the influence of light reflection. This threshold is a threshold for the brightness intensity of the image, and can be set primarily to target the remaining area excluding the area with light reflection.

[0147] The binarization step may be a step that binarizes the image based on a set threshold. Pixels above the threshold may be set to white, while those below the threshold may be set to black. This allows for the distinction between areas without light reflection and areas with reflection.

[0148] The center-surround contour merging step utilizes a technique for detecting object boundaries, potentially restoring marker information lost due to light reflection. The center-surround contour technique can detect boundaries by calculating the pixel value difference between the surrounding area and the center area. Because boundaries can sometimes become ambiguous due to light reflection, applying a high threshold value can more accurately estimate the merged boundary.

[0149] The marker restoration step may be a step of restoring a marker using an estimated boundary by merging a center surround contour. A marker is information indicating the location and outline of an object in an image, and the processor (110) can accurately extract or process an object in an image by utilizing a marker that has been restored robustly to light reflection.

[0150] The aforementioned restoration method minimizes the influence of light reflection and allows for more accurate extraction of necessary information from the image through marker restoration.

[0151] Referring to FIGS. 1 and 8, the processor (110) can generate a marker restoration image by performing binarization by merging a center surround contour that places weight on the center of the image while increasing a threshold for brightness intensity based on a light reflection compensation binary code image.

[0152] The processor (110) can generate shape information of the matrix code (300) by determining whether to display the color of each cell constituting the binary code image (222) based on the marker restoration image.

[0153] At least one component may be added or deleted in response to the performance of the components described above. Furthermore, those skilled in the art will readily understand that the relative positions of the components may be altered in response to the performance or structure of the system.

[0154] Figure 9 is a flowchart of a control method for an electronic shelf labeling system according to one embodiment. This is merely a preferred embodiment for achieving the purpose of the present invention, and it is understood that certain components may be added or deleted as needed.

[0155] Referring to FIG. 9, the camera (130) can capture the front of the shelf to obtain an analysis target image (200) (1001).

[0156] The processor (110) can extract a product image (210) and an electronic shelf label image (220) included in the analysis target image (200) through a first machine learning model (171) based on the analysis target image (200) (1002).

[0157] The processor (110) generates an upscaled image by quadrupling the size of the electronic shelf label image (220) to generate a high-resolution image through a second machine learning model (172) based on a deep residual network structure based on the electronic shelf label image (220), and samples the surrounding areas of each pixel of the upscaled image based on the upscaled image to measure the similarity between each pixel and the surrounding area, calculates a weight for each pixel based on the similarity between each pixel and the surrounding area, and adjusts each pixel value of the upscaled image based on the weight for each pixel to generate a noise-removed image, which is an image from which noise has been removed (1003).

[0158] The processor (110) can determine the positions of each corner of the display unit (121) in the electronic shelf label image (220) based on the difference between the color of the identifier area (301) included in the noise removal image and the color of the bezel (122). Based on the positions of each corner of the display unit (121), the processor (110) can extract a display unit image (221) with each corner as a vertex in the electronic shelf label image (220) (1004).

[0159] The processor (110) can extract a binary code image (222), which is an image of an area excluding the identifier area (301) from the display image (221) (1005).

[0160] The processor (110) can generate a light reflection compensation binary code image (1006) based on the binary code image (222) by a filtering method that normalizes the light exposed in the binary code image (222).

[0161] The processor (110) can generate shape information of the matrix code (300) by determining whether to display the color of each cell constituting the binary code image (222) based on the light reflection compensation binary code image (1007).

[0162] The processor (110) can control the database memory (140) in which product information is stored so that the shape information of the matrix code (300) in which the color display of each cell is determined is matched with the product information corresponding to the product image (210) (1008).

[0163] The machine learning unit (150) may be any one processor (110) among the plurality of processors (110) included in the electronic shelf label labeling system (100). Alternatively, the machine learning unit (150) may include any one processor (110) among the plurality of processors (110) included in the electronic shelf label labeling system (100).

[0164] The method for controlling the electronic shelf label labeling system (100) according to the embodiments of the present invention described so far and the embodiments to be described in the future can be implemented in the form of a program that can be driven by a processor (110).

[0165] Here, the program may include program commands, data files, and data structures, either singly or in combination. The program may be designed and produced using machine language code or high-level language code. The program may be specifically designed to implement a method for controlling the electronic shelf label labeling system (100) described above, or may be implemented using various functions or definitions that are known and available to those skilled in the art of computer software. The program for implementing the control method of the electronic shelf label labeling system (100) described above may be recorded on a recording medium readable by the processor (110). In this case, the recording medium may be a memory (170) included in the electronic shelf label labeling system (100).

[0166] The memory (170) can store a program that performs the operations described above and the operations described below, and the memory (170) can execute the stored program. In the case where there are multiple processors (110) and memories (170), they can be integrated into one chip or provided in physically separate locations. The memory (170) can include volatile memory such as Static Random Access Memory (S-RAM) and Dynamic Random Access Memory (D-RAP) for temporarily storing data. In addition, the memory (170) can include nonvolatile memory such as Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), and Electrically Erasable Programmable Read Only Memory (EEPROM) for long-term storage of control programs and control data.

[0167] The processor (110) may include various logic circuits and operation circuits, process data according to a program provided from the memory (170), and generate a control signal according to the processing result.

[0168] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. Processor; An electronic shelf label (ESL) mounted on the front of a shelf configured to allow products to be displayed in preset rows by type based on the front, and having a matrix code, which is a two-dimensional barcode capable of containing information on products displayed on the shelf; and A camera configured to capture a front view of the shelf and obtain an analysis target image including products displayed on the shelf and electronic shelf labels mounted at positions corresponding to the products, The above processor: Based on the above analysis target image, the product image and electronic shelf label image included in the analysis target image are extracted through the first machine learning model; and An electronic shelf label labeling system that controls a database memory in which product information is stored so that, based on the electronic shelf label image, the shape information of the matrix code of the electronic shelf label matches the product information corresponding to the product image.

2. In paragraph 1, The above processor: Based on the electronic shelf label image, determining whether to display the color of each cell constituting the matrix code to generate shape information of the matrix code; and An electronic shelf label labeling system that controls a database memory in which product information is stored so that the shape information of the matrix code, in which color display is determined for each of the above cells, matches the product information corresponding to the product image.

3. In paragraph 2, The above electronic shelf label is: A display unit configured to display a matrix code consisting of cells arranged along a preset number of rows and a preset number of columns; and Including a bezel configured to surround the above display portion, The above display part, An electronic shelf label labeling system configured to display an identifier area of ​​a color different from the color of the bezel at one end and the other end of the matrix code.

4. In paragraph 3, The above processor: Based on the difference between the color of the identifier area included in the electronic shelf label image and the color of the bezel, the positions of each corner of the display portion in the electronic shelf label image are determined; Based on the positions of each corner of the display unit, a display unit image is extracted from the electronic shelf label image with each corner as a vertex; Extracting a binary code image, which is an image of an area excluding the identifier area from the above display image; and An electronic shelf label labeling system that generates shape information of the matrix code by determining whether to display the color of each cell constituting the binary code image based on the binary code image.

5. In paragraph 4, The above display part, An electronic shelf label labeling system that displays all cells of a column at one end and all cells of a column at the other end as identifier areas by causing all cells of a column at one end and all cells of a column at the other end to display a color different from the color of the bezel.

6. In paragraph 5, The above display: Displays a matrix code consisting of cells arranged along four rows and six columns; In the above matrix code, all cells arranged in the 1st and 6th columns are displayed with a color different from the color of the bezel as the identifier; The above processor: Extracting images of cells arranged in columns 2, 3, 4 and 5 of the matrix code from the image of the display unit as the binary code image; and An electronic shelf label labeling system that generates shape information of the matrix code by determining whether to display colors of cells arranged in columns 2, 3, 4, and 5 in the matrix code.

7. In paragraph 5, The above processor, An electronic shelf label labeling system that generates an upscaled image by quadrupling the size of the electronic shelf label image to generate a high-resolution image through a second machine learning model based on a deep residual network structure based on the electronic shelf label image.

8. In paragraph 7, The above processor: Based on the above upscaled image, the surrounding areas of each pixel of the above upscaled image are sampled and the similarity between each pixel and the surrounding area is measured; Calculate a weight for each pixel based on the similarity between each pixel and the surrounding area; Generating a noise-removed image, which is an image with noise removed, by adjusting each pixel value of the upscaled image based on the weight for each pixel; and An electronic shelf label labeling system that determines the positions of each corner of the display portion in the electronic shelf label image based on the difference between the color of the identifier area included in the noise-removed image and the color of the bezel.

9. In paragraph 4, The above processor: Based on the binary code image, a light reflection correction binary code image is generated by a filtering method that normalizes the light exposed in the binary code image; and An electronic shelf label labeling system that generates shape information of the matrix code by determining whether to display the color of each cell constituting the binary code image based on the above light reflection correction binary code image.

10. In paragraph 9, The above processor: Applying a log transformation to the above binary code image to generate a log transformed image; A Fourier transform image representing a frequency domain is generated by Fourier transforming the above log transform image; Separate the part expressed in the high frequency region from the above Fourier transform image as an illumination element, and separate the part expressed in the low frequency region as a reflectance element; and In the above Fourier transform image, the lighting element is adjusted and the reflection element is adjusted to generate a detail-enhancing image, The above lighting element represents the intensity and distribution of lighting applied to the image, which indicates the brightness fluctuation of the image and the color tone change due to the lighting conditions. The above reflective element is an electronic shelf label labeling system that represents the characteristics of the image independent of the brightness fluctuations of the image as the color and detail of the image.

11. In paragraph 10, The above processor: Generating an inverse discrete Fourier transform image by performing an inverse discrete Fourier transform on the above detailed information-emphasized image; and An electronic shelf label labeling system that generates the light reflection compensation binary code image by applying an exponential transform to the inverse discrete Fourier transform image.

12. In paragraph 9, The above processor: Based on the above light reflection correction binary code image, a marker restoration image is generated by performing binarization by merging a center surround contour that places weight on the center of the image while increasing the threshold for brightness intensity; and An electronic shelf label labeling system that generates shape information of the matrix code by determining whether to display the color of each cell constituting the binary code image based on the marker restoration image.

13. In paragraph 1, An electronic shelf label labeling system further comprising a machine learning unit configured to learn the first machine learning model through a machine learning method by photographing the front of a learning shelf, setting a learning image including learning products displayed on the learning shelf and learning electronic shelf labels mounted at locations corresponding to the learning products as input variables, and setting an area where the learning products are located and an area where the learning electronic shelf labels are located in the learning image as output variables.

14. In paragraph 1, The above processor, An electronic shelf label labeling system that controls a database memory in which product information is stored so that, based on the electronic shelf label image, the shape information of the matrix code of the electronic shelf label matches at least one of product type information, product price information, product arrival time information, and product weight information corresponding to the product image.

15. In paragraph 2, The above processor: An electronic shelf label labeling system that controls the database memory to classify and store information on a product corresponding to the product image and shape information on a matrix code for which matching is completed as a matching completed matrix code.

16. In paragraph 15, The above camera, It is configured to photograph the front of the inspection target shelf to determine whether the displayed product is misdisplayed, and to obtain an inspection target image including the inspection target product displayed on the inspection target shelf and the electronic shelf label mounted at a position corresponding to the inspection target product. The above processor, An electronic shelf label labeling system that determines whether the shape information of the matrix code generated based on the inspection target image is identical to the matching completed matrix code corresponding to the inspection target product.

17. In paragraph 16, Further comprising a system communication unit configured to be able to communicate with the terminal of the manager managing the above shelf, The above processor: If the shape information of the matrix code generated based on the inspection target image is different from the matching completed matrix code corresponding to the inspection target product, misdisplay information is generated to the effect of re-arranging the product display on the inspection target shelf; and An electronic shelf label labeling system that controls the system communication unit to transmit the above-mentioned misprint information to the administrator's terminal.

18. In paragraph 16, The above electronic shelf label is, A label communication unit configured to receive a signal generated by the processor, The above processor: If the shape information of the matrix code generated based on the inspection target image is different from the matching completion matrix code corresponding to the inspection target product, a code change signal is generated, which is a signal for controlling the electronic shelf label of the inspection target shelf so that the electronic shelf label of the inspection target shelf displays the matching completion matrix code; and An electronic shelf label labeling system that transmits the above code change signal to the electronic shelf label of the shelf to be inspected.

19. A control method for an electronic shelf label labeling system including an electronic shelf label, a camera and a processor, which is mounted on the front of a shelf configured so that products can be displayed in rows preset by type based on the front, and which displays a matrix code, which is a two-dimensional barcode capable of containing information on products displayed on the shelf, A step of capturing a front view of the shelf by the camera to obtain an analysis target image including products displayed on the shelf and electronic shelf labels mounted at positions corresponding to the products; A step of extracting a product image and an electronic shelf label image included in the analysis target image through a first machine learning model based on the analysis target image by the processor; A step of generating shape information of the matrix code by determining whether to display the color of each cell constituting the matrix code based on the electronic shelf label image by the processor; and A control method of an electronic shelf labeling system, comprising a step of controlling a database memory in which product information is stored so that shape information of a matrix code, in which color display is determined for each cell, is matched with product information corresponding to the product image by the processor.

20. A non-transitory recording medium storing a computer program readable by a computer to execute the control method of the electronic shelf label labeling system of Article 19.

Images