Shelf monitoring device, shelf monitoring method, and shelf monitoring program

Abstract

Accurately monitor the display status of items. [Solution] The shelf monitoring device comprises a setting unit that acquires video data including display shelves and items and sets a monitoring area on the display shelves where the items are displayed; a detection unit that uses a trained model to recognize the background of the items in the monitoring area as a background image and gradually detects the inventory level of the items based on the area proportion of the background image; and an output unit that outputs information indicating the inventory level of the items.

Description

[Technical field]

[0001] The present disclosure relates to a shelf monitoring device, a shelf monitoring method, and a shelf monitoring program. [Background technology]

[0002] In the retail industry, for example, the efficiency of operations is being improved by systematizing the management of product inventory and / or sales. For inventory management, for example, a system that monitors product inventory based on images of product display areas (e.g., display shelves) captured by surveillance cameras is sometimes used. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-103153 A [Patent Document 2] Patent No. 6112436 Summary of the Invention [Problem to be solved by the invention]

[0004] However, for example, if the shelf levels to be monitored (in other words, the monitoring area or monitoring range) are not appropriately set for footage that includes display shelves, it may be difficult to accurately monitor product inventory (e.g., shortages or missing items).

[0005] As a result, there are cases where a person has no choice but to visually check the stock of each product. When a person visually checks the stock, the number of individual products actually displayed in the display area (actual stock number) may not match the number of individual products managed by, for example, an inventory management system (theoretical stock number).

[0006] If the actual inventory quantity and the theoretical inventory quantity do not match, the inventory management data must be manually revised, for example by correcting the theoretical inventory quantity to match the actual inventory quantity confirmed by visual inspection, which takes time.

[0007] Performing such correction work every day and / or every hour is a burden for the inventory manager, and so, for example, cases may occur where a particular product is left out of stock. If a product is left out of stock, opportunities to sell the product are lost and, in some cases, this can lead to a negative image among customers.

[0008] Non-limiting examples of the present disclosure contribute to providing a shelf monitoring device, a shelf monitoring method, and a shelf monitoring program that can accurately monitor the display status of items by appropriately setting a monitoring area for an image that includes display shelves. [Means for solving the problem]

[0009] A shelf monitoring device according to one embodiment of the present disclosure includes a setting unit that acquires video data including display shelves and items, and sets a monitoring area in which the items are displayed on the display shelves, a detection unit that uses a trained model for recognizing the background of the items in the monitoring area as a background image to gradually detect the inventory level of the items based on the area proportion of the background image, and an output unit that outputs information indicating the inventory level of the items.

[0010] In addition, a shelf monitoring method according to one aspect of the present disclosure includes a computer acquiring video data including display shelves and items, setting a monitoring area in which the items are displayed on the display shelves, and using a trained model for recognizing the background of the items in the monitoring area as a background image, the computer gradually detects the inventory level of the items based on the area proportion of the background image, and outputs information indicating the inventory level of the items.

[0011] In addition, a shelf monitoring program according to one embodiment of the present disclosure causes a computer to execute a process of acquiring video data including display shelves and items, setting a monitoring area in which the items are displayed on the display shelves, using a trained model for recognizing the background of the items in the monitoring area as a background image, gradually detecting the inventory level of the items based on the area proportion of the background image, and outputting information indicating the inventory level of the items.

[0012] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0013] According to one aspect of the present disclosure, a monitoring area can be appropriately set for an image including a display shelf, so that the display status of items can be accurately monitored.

[0014] Further advantages and benefits of certain aspects of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing a configuration example of an inventory management system according to an embodiment. [Diagram 2] 1 is a schematic front view of a display shelf with shelf labels according to one embodiment; [Diagram 3] FIG. 3 is a partially enlarged front view of the display shelf shown in FIG. [Figure 4] FIG. 1 is a diagram showing an example of shelf allocation information according to an embodiment; [Diagram 5] A block diagram showing an example of the configuration of the computer shown in FIG. [Figure 6] A flowchart showing an example of the operation of the computer (inventory monitoring device) illustrated in FIGS. 1 and 5. [Figure 7] FIG. 6 is a diagram showing an example of shelf tags being detected by image recognition in the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 8] FIG. 6 is a diagram showing an example of how a monitoring area is set by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 9] FIG. 6 is a diagram showing an example of a flow of shelf label association processing by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 10] FIG. 6 is a diagram showing an example of a flow of shelf label association processing by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 11] FIG. 6 is a diagram showing an example of a flow of shelf label association processing by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 12] FIG. 6 is a diagram showing an example of a flow of shelf label association processing by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 13] FIG. 6 is a diagram showing an example of a flow of shelf label association processing by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 14] FIG. 6 is a diagram showing an example of setting a monitoring area by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 15A] FIG. 6 is a diagram showing an example of a search sequence in a shelf label detection process performed by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 15B] FIG. 6 is a diagram showing an example of a search sequence in a shelf label detection process performed by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 16] FIG. 6 is a diagram showing an example of setting a monitoring area using shelf allocation information by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 17] FIG. 6 is a diagram showing an example of correction of shelf tag detection results by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 18] FIG. 6 is a diagram showing an example of correction of shelf tag detection results by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 19] FIG. 6 is a diagram showing an example of correction of shelf tag detection results by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 20]FIG. 2 is a diagram for explaining an example in which a PTZ (pan, tilt, and zoom) camera is applied to the camera illustrated in FIG. 1; [Figure 21] FIG. 21 shows an example in which the recognition model used for pattern matching in image recognition differs for each shooting direction in FIG. [Figure 22A] FIG. 6 shows an example of setting a tiered inventory level using the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 22B] FIG. 6 shows an example of setting a tiered inventory level using the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 22C] FIG. 6 shows an example of setting a tiered inventory level using the inventory monitoring device illustrated in FIGS. 1 and 5. [Diagram 23] FIG. 6 is a diagram showing an example of how an inventory level in the depth direction of a shelf is detected in the inventory monitoring device illustrated in FIGS. 1 and 5 . [Figure 24] FIG. 6 is a diagram showing an example of how an inventory level in the depth direction of a shelf is detected in the inventory monitoring device illustrated in FIGS. 1 and 5 . [Figure 25A] FIG. 6 is a diagram showing an example of a moving object removal process performed by the inventory monitoring device illustrated in FIGS. 1 and 5. [Figure 25B] FIG. 6 is a diagram showing an example of a moving object removal process performed by the inventory monitoring device illustrated in FIGS. 1 and 5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or duplicate description of substantially the same configuration may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

[0017] It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0018] (System configuration example) Fig. 1 is a block diagram showing an example of the configuration of an inventory management system according to an embodiment. The inventory management system 1 shown in Fig. 1 may include, for example, a camera 10 and a computer 20. Note that the "inventory management system" may be rephrased as an "inventory monitoring system."

[0019] The camera 10 is installed, for example, in a store that sells products, and captures video including an area where the products are displayed, for example, an area where product display shelves 50 are installed. There are no limitations on the type or business format of the store in which the display shelves 50 are installed, and the types of products that the store handles.

[0020] For example, the display shelf 50 may be installed in a supermarket, a convenience store, a department store, a mass retailer, a discount store, or a shop or a sales booth (or a sales corner) installed in various facilities. The display shelf 50 may be installed outdoors, not limited to indoors.

[0021] The camera 10 may be a dedicated camera for photographing the area including the display shelves 50, or may be a camera that is also used for other purposes or applications, such as a security camera. Furthermore, a plurality of cameras 10 may be provided in the inventory management system 1.

[0022] The objects to be photographed by the camera 10 (which may also be referred to as "monitoring objects") and the display shelves 50 may correspond one-to-one, one-to-many, or many-to-one. For example, one camera 10 may photograph one display shelf 50, or one camera 10 may photograph multiple display shelves 50.

[0023] For example, by using a camera with a variable shooting direction and / or angle of view, such as a PTZ camera, as the camera 10, it is possible to include a plurality of different display shelves 50 in the shooting targets with one PTZ camera 10. Alternatively, it is possible to include different areas or spaces of one display shelf 50 in the shooting targets with one PTZ camera 10.

[0024] For example, if it is difficult to capture the width or height of a single display shelf 50 in the camera image using a single camera 10, the shooting direction of one or more PTZ cameras 10 can be variably controlled to capture multiple areas or spaces with different widths or heights in the camera image.

[0025] In this way, by using a PTZ camera as the camera 10, it is not necessary to install a camera 10 for each display shelf 50 or for each different area or space of the display shelf 50, thereby reducing the number of cameras 10 installed in the inventory management system 1.

[0026] The computer 20 illustrated in Fig. 1 is an example of an information processing device, and may be a personal computer (PC) or a server. The server may include a cloud server. The computer 20 and the camera 10 are connected to each other, for example, by wire or wirelessly, and communicate with each other. The computer 20 and the camera 10 may also be connected via a network.

[0027] The "network" may be a wired network or a wireless network. Examples of wired networks include an intranet, the Internet, and a wired LAN (Local Area Network). An example of a wireless network is a wireless LAN.

[0028] The computer 20 receives, for example, video data captured by the camera 10 (hereinafter sometimes abbreviated as "camera video") and analyzes the received camera video. For example, the computer 20 monitors the inventory of products on the display shelves 50 by performing video recognition on the camera video of the display shelves 50, and detects shortages or missing items.

[0029] Note that "video recognition" may be rephrased as "image recognition." Furthermore, for convenience, detecting shortages or out-of-stock items may be collectively referred to as "out-of-stock detection." "Detection" may be rephrased as "detection." Furthermore, the source of the camera video may be the camera 10, or may be, for example, a recording device that records the video data captured by the camera 10.

[0030] Monitoring the inventory of products may include detecting the positions of shelf labels attached to the display shelves 50. The shelf labels may display information about the products (hereinafter referred to as "product information"), such as the product name and / or price. The shelf labels may be paper shelf labels or electronic shelf labels. The electronic shelf labels may be composed of a liquid crystal display or the like, or may be composed of electronic paper or the like. Furthermore, the electronic shelf labels may be equipped with wireless communication functions or the like, and may be capable of remotely rewriting the information presented. The "shelf labels" may be called by other names, such as shelf tags, shelf cards, or shelf labels.

[0031] The computer 20 may detect the positions of the shelf tags attached to the display shelves 50, for example, by image recognition of camera images, and set an area or space on the display shelves 50 for monitoring inventory based on the detected shelf tag positions.

[0032] Hereinafter, the area or space for which the inventory of products is monitored may be collectively referred to as a “monitoring area” or a “monitoring range.” An example of setting the monitoring area for the display shelf 50 based on the positions of the shelf tags 51 detected by video recognition will be described later.

[0033] Fig. 2 shows a schematic front view of a display shelf 50 with shelf tags 51 attached. Fig. 3 shows a partially enlarged front view of the display shelf 50 shown in Fig. 2. As a non-limiting example, Figs. 2 and 3 show an embodiment in which the display space of the display shelf 50 is divided into four spaces in the height direction of the display shelf 50 by three shelves 52. In Figs. 2 and 3, the area surrounded by a dotted line frame represents a situation in which a product 70 is out of stock.

[0034] FIG. 3 also shows an example in which image data of display shelf 50 captured by camera 10 is recognized by computer 20, and if a shortage or missing item is detected in a monitoring area set based on the position of the detected shelf tag 51, that information is output.

[0035] The display space divided in the height direction in the display shelf 50 may be referred to as a "shelf level." Fig. 2 shows an example focusing on two shelf levels. Note that there is no limitation on the way the display space in one display shelf 50 is divided. The display space may be divided not only in the height direction of the display shelf 50, but also in the width direction of the display shelf 50.

[0036] The shelf labels 51 may be attached to any position on the display shelf 50 where the correspondence with the displayed products 70 can be visually confirmed. For example, the shelf labels 51 may be attached to the shelves 52. The products 70 are displayed, for example, in an area or space corresponding to the position of the corresponding shelf label 51 in each display space (the space above the shelf label 51 in the examples of FIGS. 2 and 3).

[0037] In setting the monitoring area and / or monitoring the inventory, information on shelf allocation (hereinafter referred to as "shelf allocation information") may be used in addition to the position information of the detected shelf tag 51. "Shelf allocation" refers to, for example, planning how many of what products are to be displayed (or allocated) in which display space of the display shelf 50.

[0038] Fig. 4 shows an example of the shelf allocation information 400. As shown in Fig. 4, the shelf allocation information 400 may include, for example, information indicating the positions at which the products 70 are displayed and information about the products 70 displayed at those positions. Note that, hereinafter, the information indicating the positions at which the products 70 are displayed may be referred to as "display position information," and the information about the products 70 may be referred to as "product information."

[0039] As a non-limiting example, the display position information may include information indicating one or more of a store number, a floor number, an aisle number, a shelf number, a shelf level number, and a display position on a shelf level.

[0040] The product information may include information capable of specifying or identifying each product 70, such as the type or content of the product 70. Non-limiting examples of information capable of specifying or identifying the product 70 include a product name such as "XXX pasta" or "YYY curry" or a product code.

[0041] The product information may include, for example, information indicating the size of the product 70 (at least one of width, height, and depth), or information indicating the number of products 70 to be displayed, in other words, information indicating the number of products to be displayed. The product information may also include information indicating the price of the product 70.

[0042] The "number of displayed products" may indicate, for example, the number of displayed products 70 in one or more of the width, height, and depth directions of the shelf. Based on one or both of the information indicating the size of the products 70 and the information indicating the number of displayed products 70, for example, it is possible to improve the accuracy of identifying the space or area occupied by a plurality of products 70 on the shelf.

[0043] Therefore, by referring to the shelf allocation information 400, the computer 20 can improve the accuracy of setting the monitoring area based on the shelf label position, and as a result, the accuracy of detecting a shortage of the product 70 can be improved.

[0044] Furthermore, the computer 20 may correct the detection result (e.g., shelf label position) of the shelf label 51 attached to the display shelf 50 based on the image recognition, based on the shelf allocation information 400. Correcting the shelf label position may include, for example, correcting an oversight of detection of the shelf label 51 by image recognition, based on the shelf allocation information 400.

[0045] An example of setting a monitoring area and an example of correcting a shelf label position using the shelf allocation information 400 will be described later.

[0046] If the shelf allocation information 400 is not used, specific information such as the type of product 70 that is in short supply or out of stock in the monitoring area cannot be identified, but it is possible to detect which shelf and which position of the product 70 is in short supply or out of stock from the results of image recognition of the monitoring area set based on the shelf label position (for example, the proportion of the background image appearing in the monitoring area). By using the shelf allocation information 400 in combination with such detection processing, it becomes possible to identify specific information such as the type of product 70 that is involved in the detected shortage. For example, it becomes possible to present information such as "XXX pasta is out of stock" or "XXX pasta in row △ on row XX is out of stock."

[0047] (Example of computer 20 configuration) Next, a configuration example of the computer 20 will be described with reference to Fig. 5. As shown in Fig. 5, the computer 20 may include, for example, a processor 201, an input device 202, an output device 203, a memory 204, a storage 205, and a communication unit 206.

[0048] The processor 201 controls the operation of the computer 20. The processor 201 is an example of a circuit or device having a computing capability. The processor 201 may be, for example, at least one of a central processing unit (CPU), a micro processing unit (MPU), and a graphics processing unit (GPU).

[0049] The input device 202 may include, for example, at least one of a keyboard, a mouse, an operation button, and a microphone. Data or information may be input to the processor 201 through the input device 202.

[0050] The output device 203 may include, for example, at least one of a display (or monitor), a printer, and a speaker. The display may be, for example, a touch panel display. The touch panel display may be considered to correspond to both the input device 202 and the output device 203.

[0051] The memory 204 stores, for example, a program executed by the processor 201, and data or information processed in response to the execution of the program. The memory 204 may include a random access memory (RAM) and a read only memory (ROM). The RAM may be used as a working memory for the processor 201. The "program" may also be referred to as "software" or an "application."

[0052] The storage 205 stores the program executed by the processor 201, and data or information processed in response to the execution of the program. The above-mentioned shelf allocation information 400 may be stored in the storage 205. The shelf allocation information 400 may be stored in the storage 205 in advance, or may be provided from a shelf allocation system (not shown) that manages the shelf allocation information 400 and stored in the storage 205, for example.

[0053] Storage 205 may include a solid-state drive device, such as a hard disk drive (HDD) or a solid-state drive (SSD). In addition to or in place of a solid-state drive device, non-volatile memory, such as flash memory, may be included in storage 205.

[0054] The programs may include an inventory monitoring program that monitors the inventory of the products 70 by image recognition as described above. All or a part of the program code constituting the inventory monitoring program may be stored in the memory 204 and / or the storage 205, or may be incorporated into a part of the operating system (OS).

[0055] The program and / or data may be provided in a form recorded on a recording medium readable by the computer 20. Examples of the recording medium include a flexible disk, a CD-ROM, a CD-R, a CD-RW, an MO, a DVD, a Blu-ray disk, a portable hard disk, etc. Another example of the recording medium is a semiconductor memory such as a USB (Universal Serial Bus) memory.

[0056] Furthermore, the program and / or data may be provided (downloaded) to the computer 20 from a server (not shown), for example, via a communication line. For example, the program and / or data may be provided to the computer 20 via the communication unit 206 and stored in the memory 204 and / or the storage 205. Furthermore, the program and / or data may be provided to the computer 20 via the input device 202 and stored in the memory 204 and / or the storage 205.

[0057] The communication unit 206 includes, for example, a communication interface (IF) 261 for communicating with the camera 10. The communication IF 261 may be either a wired interface or a wireless interface.

[0058] For example, the communication IF 261 receives video data captured by the camera 10. The received video data is stored in the memory 204 and / or the storage 205, for example, via the processor 201. If the camera 10 is a PTZ camera, the communication unit 206 may control the shooting direction and / or the angle of view of the PTZ camera 10 by communicating with the PTZ camera 10 in response to an instruction from the processor 201, for example.

[0059] The communication unit 206 may also include a communication IF 262 for communicating with "other computers" (not shown) different from the computer 20. The "other computers" may be, for example, servers connected to a wired or wireless network, or user terminals connected to a wired or wireless network. The "other computers" may correspond to the computers in the shelf allocation system described above.

[0060] The user terminal may be owned, for example, by an inventory manager of the product 70. Non-limiting examples of the user terminal include a PC, a mobile phone (including a smartphone), and a tablet terminal. Information related to inventory management or inventory monitoring of the product may be provided to the user terminal.

[0061] The processor 201 can, for example, read and execute an inventory monitoring program stored in the memory 204 and / or the storage 205, causing the computer 20 to function as an inventory monitoring device that monitors the inventory of the product 70 through image recognition.

[0062] For example, when the processor 201 executes the inventory monitoring program, an inventory monitoring device 20 including a shelf label detection unit 211, a monitoring area setting unit 212, a stockout detection unit 213, and an output unit 214, as illustrated in Fig. 5, is embodied. Optionally, in the inventory monitoring device 20, one or both of a shelf label association unit 215 and a detected shelf label correction unit 216 may be embodied in response to the execution of the inventory monitoring program.

[0063] The shelf label detection unit 211 detects the shelf labels 51 included in the camera image, for example, by image recognition of the camera image that includes all or part of the display shelf 50. For example, the shelf label detection unit 211 may detect the shelf labels 51 by performing pattern matching on the camera image using a template image that corresponds to the shape and / or color of the shelf label 51.

[0064] The monitoring area setting unit 212 sets a monitoring area for the display shelf 50 based on the positions of the shelf labels 51 detected by the shelf label detection unit 211, for example.

[0065] The stock-out detection unit 213 is an example of a monitoring unit, and monitors the inventory (or “display status”) of products 70 on the display shelf 50 based on, for example, changes in the image corresponding to the presence or absence of products in the monitoring area set by the monitoring area setting unit 212.

[0066] For example, the shortage detection unit 213 may detect a shortage or shortage of product 70 by using a background image that appears in the monitored area when product 70 is insufficient or out of stock as a template image and performing pattern matching on the camera image of the monitored area.

[0067] The shape of the shelf labels 51 in the camera images may change depending on the installation position and / or shooting direction of the camera 10. For example, the shape of the shelf labels 51 differs between a camera image capturing the display shelf 50 from the front and a camera image capturing the display shelf 50 from an oblique direction shifted from the front.

[0068] Furthermore, depending on the installation position and / or shooting direction of the camera 10 (hereinafter, for convenience, these may be collectively referred to as "camera position"), the background image that appears in the camera image when the product 70 is insufficient or out of stock may also change.

[0069] For example, in a camera image capturing the display shelf 50 from the front, an image in the front direction of the back panel located at the rear of the display shelf 50 may correspond to the background image. Also, in a camera image capturing the display shelf 50 from diagonally above, for example, the surface of the shelf board 52 on which the products 70 are displayed may correspond to the background image. In a camera image capturing the display shelf 50 from an oblique side, for example, the surface of a partition board (not shown) that separates the shelf levels in the left and right direction may correspond to the background image.

[0070] In this way, since the shape of the shelf label 51 and / or the background image change depending on the camera position, a template image (which may also be called a "recognition model") used for pattern matching in image recognition may be prepared corresponding to the camera position. Note that in this embodiment, the recognition model is a template image, and the shape of the shelf label and / or the background image is described as being recognized by pattern matching, but other implementation methods are also possible. For example, a trained model generated by performing machine learning on each of the shelf label and / or the background image may be used as the recognition model to recognize the shelf label and / or the background image.

[0071] For example, multiple template images may be prepared when the camera 10 is installed in multiple locations and / or when the shooting direction is variable like the PTZ camera 10. The template images are stored in the storage 205, for example, and are read out by the processor 201 at appropriate times.

[0072] The output unit 214 is an example of a notification information generating unit that generates and outputs information to be presented (e.g., notified) to an inventory manager. The output unit 214 generates notification information including, for example, the detection result by the stockout detection unit 213 and / or information based on the detection result, and outputs the notification information to the output device 203 and / or the communication unit 206.

[0073] As a non-limiting example, information notifying the detection of a shortage of product 70 (which may be referred to as “notification information,” “shortage information,” or “alert information”) may be output to a display and / or a printer, which are examples of output device 203.

[0074] The output unit 214 may generate notification information based on information in which the shelf allocation information 400 is associated with either or both of the detection result by the stockout detection unit 213 and the detection result by the shelf label detection unit 211.

[0075] For example, based on information correlating the detection result by the out-of-stock detection unit 213 with the shelf allocation information 400, the output unit 214 can generate notification information including the location (out-of-stock area) related to the out-of-stock detection of the product 70 and / or the product name of the product 70 detected to be out of stock.

[0076] The notification information may be transmitted to the "other computer" via the communication unit 206. The notification information may be transmitted via the communication unit 206 by e-mail.

[0077] The shelf label associating unit 215 associates, for example, each of the shelf labels 51 detected by the shelf label detecting unit 211 with shelf allocation information 400 (for example, product information).

[0078] For example, when the shelf allocation information 400 includes information on the size and number of products 70 displayed as an example of product information, the detected shelf label correction unit 216 may correct the shelf label position detected by the shelf label detection unit 211 based on the product information.

[0079] 5 is merely an example. The hardware and / or functional blocks may be increased or decreased as appropriate in the inventory monitoring device 20. For example, the hardware and / or functional blocks may be added, deleted, divided, or integrated in the inventory monitoring device 20 as appropriate.

[0080] (Example of operation) Next, an example of the operation of the inventory monitoring device 20 will be described. 6, the inventory monitoring device 20, for example, acquires camera footage (S11) and detects shelf tags 51 included in the camera footage by analyzing the acquired camera footage, for example, in the shelf tag detection unit 211 (S12). For example, as shown by the thick frame 500 in FIG. 7, eight shelf tags 51 are detected by video recognition.

[0081] 7, two shelf labels 51 are detected on the mth shelf board 52 from the bottom, and three shelf labels 51 are detected on each of the shelf board 52 located above that (the m+1th shelf board from the bottom) and the shelf board 52 located even higher (the m+2th shelf board from the bottom), where m is an integer of 1 or more.

[0082] The positions of two shelf labels 51 detected on the m-th shelf board 52 from the bottom are, for example, the n-th and n+1-th positions (n ​​is an integer equal to or greater than 1) from the left. Similarly, the positions of three shelf labels 51 detected on the m+1-th shelf board 52 from the bottom are, for example, the n-th, n+1-th, and n+2-th positions from the left. The positions of three shelf labels 51 detected on the m+2-th shelf board 52 from the bottom are, for example, the n-th, n+1-th, and n+2-th positions from the left. The positions of "m-th row and n-th row" may be expressed as the positions of "m-th row, n-th number" or "m-th row, n-th column".

[0083] If the inventory management system 1 is equipped with multiple cameras 10, or if a PTZ camera is used as the camera 10, the inventory monitoring device 20 may, for example, acquire information indicating the camera position and provide it to the shelf label detection unit 211 (S11a in FIG. 6).

[0084] The information indicating the camera position may be associated in advance with, for example, the shelf allocation information 400. The shelf label detection unit 211 may, for example, identify which camera 10 and in which shooting direction the camera image is taken based on the information indicating the camera position and the shelf allocation information 400, and set a recognition model suitable for detecting the shelf label 51 by video recognition. Note that an example of the association between the information indicating the camera position and the shelf allocation information 400 will be described later.

[0085] In response to the detection of the shelf label 51 (S12 in FIG. 6), the inventory monitoring device 20, for example, by the monitoring area setting unit 212, sets a monitoring area for monitoring the inventory of the products 70 (S13 in FIG. 6). For example, the monitoring area setting unit 212 sets one of the detected shelf labels 51 as the reference shelf label 51. As a non-limiting example, FIG. 7 shows an example in which the shelf label 51 on the bottom left (first shelf, number 1) is set as the reference shelf label 51.

[0086] The reference shelf tag 51 may be set autonomously by the inventory monitoring device 20 (eg, the monitoring area setting unit 212) or may be specified (manually specified) by a user of the inventory monitoring device 20 (eg, an inventory manager).

[0087] For example, by making the appearance, such as the color (eg, frame color) and / or shape, of the reference shelf label 51 different from the appearances of the other shelf labels 51, the reference shelf label 51 can be autonomously set by image recognition.

[0088] Alternatively, information such as the shelf label 51 of "X shelf, △ number" may be input to the monitoring area setting unit 212 via the input device 202 as information specifying the reference shelf label 51. Also, manual specification may be used to complement the autonomous setting.

[0089] Depending on the setting of the reference shelf label 51, the monitoring area setting unit 212 may detect shelf labels 51 that are adjacent to the reference shelf label 51 in the vertical and / or horizontal directions based on, for example, the shelf allocation information 400 (e.g., display position information) illustrated in FIG. 4.

[0090] For convenience, shelf labels 51 that are adjacent in the vertical and / or horizontal directions may be referred to as “adjacent shelf labels 51.” “Detection” of an adjacent shelf label 51 may be rephrased as “searching” or “searching” for the adjacent shelf label 51.

[0091] The monitoring area setting unit 212 may set a monitoring area based on the detected distance between the shelf labels 51. An example of setting a monitoring area is shown in FIG.

[0092] In FIG. 8, for example, when the shelf label 51 at the m (=1)th row and the n (=1)th row is set as the reference shelf label, the monitoring area setting unit 213 obtains the distance between the reference shelf label 51 and the adjacent shelf label 51.

[0093] For example, the distance Rx between the reference shelf label 51 at the nth level in the mth row and the shelf label 51 at the n+1th level in the mth row to the right is detected, and the distance Ry between the reference shelf label 51 and the shelf label 51 at the nth level in the m+1th row above is detected.

[0094] The monitoring area setting unit 212 sets, for example, a monitoring area MA (see dotted line frame) whose size and shape are determined by the distances Rx and Rx, as a monitoring area for the shelf label 51 at number 1 on the first tier. For the other shelf labels 51 as well, the monitoring area setting unit 212 detects the distances Rx and Ry between adjacent shelf labels 51, and sets a monitoring area MA for each of the shelf labels 51 detected by video recognition.

[0095] As described above, for each shelf label 51 detected by video recognition, the monitoring area setting unit 212 calculates the distance between that shelf label 51 and the adjacent shelf label, and sets a monitoring area MA corresponding to each shelf label 51 based on the calculated distance. Therefore, it is possible to accurately set a monitoring area MA for the product 70 displayed corresponding to the shelf label 51.

[0096] The shape of the monitoring area MA may be rectangular or circular, such as elliptical. The shelf labels 51 may be arranged according to a predetermined rule for each space in which the products 70 are displayed. For example, the shelf labels 51 may be arranged according to a certain rule, such as placing the shelf labels 51 at the lower left of the products 70. If a certain rule is defined, it is possible to set an accurate monitoring area MA based on the position of each shelf label 51 or the distance between the shelf labels 51 and the certain rule. For example, if the shelf label 51 is arranged at the lower left of the products 70, it is understood that the monitoring area exists to the right of the shelf label 51.

[0097] Furthermore, when there is a shelf level on which no shelf label 51 is placed (for example, the top shelf level), the distance between shelf labels 51 including the dummy shelf label 51 may be found by placing a dummy shelf label 51. When finding the distance Ry above the top shelf level using a dummy shelf label 51, for example, the dummy shelf level 51 may be placed on a "POP" placed on the shelf.

[0098] "POP" refers to an advertising medium that displays, for example, a product name, product price, a catchphrase, an explanation, and / or an illustration on paper. Alternatively, for the distance Ry on a shelf level on which no shelf label 51 is installed, the distance between the shelf labels 51 may be calculated based on the distance to other shelf levels (for example, the lower shelf level).

[0099] For example, for a shelf label 51 that does not have an adjacent shelf label 51, the distance Ry calculated for another shelf label 51 may be reused. For example, the Ry of the shelf level directly below that has less perspective distortion may be reused, or the Ry of another shelf level may be converted based on the camera parameters to estimate Ry. Alternatively, for a shelf level on which no shelf label 51 is installed, the distance between shelf labels 51 may be set manually.

[0100] Furthermore, for a shelf label 51 located at the right end of a shelf level, if there is another adjacent display shelf 50, the distance between the shelf labels 51 may be obtained as the distance between the shelf label 51 and the shelf label 51 attached to that display shelf 50. Alternatively, for a shelf label 51 located at the right end of a shelf level, the distance from the edge of the image in the camera video may be set as the distance between the shelf labels 51.

[0101] The shelf allocation information 400 may include, for example, information on one or more of the width, height, depth, and number of displays of the products 70 to be monitored. In this case, the detected shelf label correction unit 216 may correct the shelf label position detected by the shelf label detection unit 211 based on the information (S12a in FIG. 6).

[0102] In other words, the shelf allocation information 400 may be used to confirm and / or correct the accuracy of the detection result of the shelf label 51 by image recognition. By correcting the shelf label position, it is possible to improve the accuracy of detecting a shortage of the product 70 to be monitored. An example of correcting the shelf label position will be described later with reference to Figs. 17 to 19.

[0103] After setting the monitoring area MA, the inventory monitoring device 20 detects areas in the monitoring area MA where products 70 are insufficient or out of stock, for example, by pattern matching between the camera footage of each monitoring area MA and the background footage (S14 in Figure 6).

[0104] 8, for example, for the shelf tags 51 in m rows and n columns, only two products 70 are displayed in a space that can display three products 70. Therefore, a background image equivalent to one product 70 appears in the camera image of the monitoring area MA.

[0105] The stockout detection unit 213 detects that the product 70 is not displayed in the area where the background image appears by matching the background image that appears in the monitoring area MA with the template image in pattern matching. In other words, a shortage or stockout of the product 70 corresponding to the shelf tag 51 is detected.

[0106] In response to detection of a shortage or out-of-stock of the product 70, the inventory monitoring device 20 generates and outputs, for example, by the output unit 214, information notifying the detection of the out-of-stock of the product 70 (S15 in FIG. 6). The information notifying the detection of the out-of-stock of the product 70 may include text information and / or audio information, and may also include information for highlighting an area related to the detection of the out-of-stock on a display showing the camera image, for example.

[0107] Non-limiting examples of the highlighting modes include the following: The following highlighting modes may be combined as appropriate. In the camera image, the color of the area where the missing item is detected (hereinafter, for convenience, may be referred to as the "missing area") is changed to a color (highlight color) that stands out more than the colors of other areas. -Shortage areas are displayed flashing. The out-of-stock area is displayed with a solid or dotted frame. The solid or dotted frame may be colored (highlighted). - A solid or dotted frame indicating a stock-out area will be displayed flashing.

[0108] As described above, according to the inventory monitoring device 20, a monitoring area is set for the display shelves 50 in the camera image based on the position of the shelf tags 51 detected by image recognition of the camera image including the display shelves 50, so that a monitoring area can be appropriately set for the camera image including the display shelves 50 without relying on manual operation, for example.

[0109] When the reference shelf label is manually specified, it is not possible to completely eliminate manual operations when setting the monitoring area. However, since most of the setting of the monitoring area can be automated and an accurate reference shelf label can be set, the monitoring area can be set more accurately and quickly than in the past.

[0110] In other words, it is possible to automate the setting of the display space to be monitored in the display shelf 50. Therefore, even if the relative positional relationship between the display shelf 50 and the camera 10 changes due to external factors such as vibration, it is possible to re-set the appropriate monitoring area by detecting the shelf labels 51 through image recognition.

[0111] The inventory monitoring device 20 monitors the inventory of the products 70 on the display shelves 50 based on changes in the image corresponding to the presence or absence of the products 70 in the set monitoring area, and notifies the monitoring results to, for example, an inventory manager, thereby preventing cases where a specific product is left out of stock. This prevents lost sales opportunities and improves the image of the product among customers.

[0112] (Shelf label association process) As indicated by the dotted line in FIG. 6, a shelf label association process (S14a) may be performed by the shelf label association unit 215 between the detection of missing items (S14) and the output of information (S15).

[0113] The shelf label association process shown in S14a in Fig. 6 may be performed, for example, as follows: First, the shelf label association unit 215 sets one of the shelf labels 51 detected by video recognition (for example, "tier 3, no. 1" in the upper left) as a reference shelf label, as shown in Fig. 9, for example.

[0114] Next, the shelf label associating unit 215 detects, for example, as shown in Fig. 10, "the shelf label 51 having the closest x coordinate among the shelf labels 51 having close y coordinates" as an adjacent shelf label 51 based on the shelf allocation information 400. In the example of Fig. 10, the shelf label 51 "tier 3, no. 2" is detected.

[0115] The shelf label association unit 215 repeats the process of detecting adjacent shelf labels 51 illustrated in FIG. 10 until the shelf label 51 located at the end of the shelf level to which the reference shelf label 51 belongs, for example, the shelf label 51 "Level 3, No. 3," is detected.

[0116] When a shelf label 51 located at the end of the shelf tier to which the reference shelf label 51 belongs is detected, the shelf label association unit 215 detects the shelf label 51 "having the closest y coordinate" to the shelf label 51 located at the end of the shelf tier to which the reference shelf label 51 belongs. For example, as shown in Fig. 11, the shelf label 51 "tier 2, no. 3" is detected.

[0117] When a shelf label 51 located at the end of the shelf level to which the reference shelf label 51 belongs is detected, the shelf label association unit 215 may return the search reference to the reference shelf label 51 and detect the shelf label 51 with the "closest y coordinate," as shown in Fig. 12. In this case, the shelf label 51 at "tier 2, no. 1" is detected.

[0118] The shelf label association unit 215 repeats the shelf label detection process illustrated in Figures 10 and 11 or Figures 10 and 12 until all shelf labels 51 are detected. Figure 15A shows an example of a search order in the shelf label detection process illustrated in Figures 10 and 11, and Figure 15B shows an example of a search order in the shelf label detection process illustrated in Figures 10 and 12. It should be noted that the search order for the shelf labels 51 is not limited to the order illustrated in Figures 15A and 15B.

[0119] When detection of all shelf labels 51 has been completed, the shelf label associating unit 215 associates the shelf label 51 of "tier m, number n" (see FIG. 13) with the shelf allocation information 400 (for example, product information).

[0120] Through the shelf label association process, shelf allocation information 400 including product information is associated with each shelf label 51 detected by video recognition. Therefore, even if the monitoring areas MA are set based on the distance between the shelf labels 51 as described above (see FIG. 14), it is possible to identify the product information of the monitoring target in each monitoring area MA. Note that the height of the topmost monitoring area MA may be set based on the height of the lower shelf, for example.

[0121] The setting of the reference shelf label 51 may be stored, for example, in the storage 205. In the second and subsequent shelf label association processes, the stored setting of the reference shelf label 51 may be followed. Therefore, it may not be necessary to reset the reference shelf label 51.

[0122] In addition, the shelf label associating unit 215 may associate, for example, one or both of the information indicating the out-of-stock area detected by the out-of-stock detection unit 213 and the information of the shelf label 51 detected by video recognition with the shelf allocation information 400.

[0123] (Generating notification information based on shelf allocation information) 6, the output unit 214 may generate notification information based on the shelf allocation information 400. For example, the output unit 214 may generate notification information including the location (out-of-stock area) related to the detection of out-of-stock of the product 70 and / or the product name and / or product code of the product 70 whose out-of-stock has been detected, based on association information between the information indicating the out-of-stock area and the information of the detected shelf label 51. The generated notification information is output to, for example, the output device 203 and / or the communication unit 206.

[0124] In this way, by using the shelf allocation information 400, the inventory monitoring device 20 can notify, for example, an inventory manager, of information including the product name and / or product code of each product 70 related to the detected stockout.

[0125] Therefore, even if it is difficult to identify individual product names by analyzing the camera images, the inventory manager can accurately grasp, for example, which shelf level is out of stock of which product 70. Therefore, the inventory manager can accurately and smoothly replenish products 70 on each shelf level, for example.

[0126] (Example of setting monitoring areas using shelf allocation information) Next, an example of setting the monitoring area MA using information indicating the size of the product 70 and the number of products 70 displayed in the shelf allocation information 400 will be described.

[0127] For example, the shelf allocation information 400 includes information that "three products 70 each having width X and height Y are displayed on shelf label 51 at shelf level m and shelf level n." In this case, the monitoring area setting unit 212 may set a monitoring area MA of a size (width X and height Y) that includes three products 70, as shown in Fig. 16. Note that in the example of Fig. 16, the size of the products 70 in the depth direction may be arbitrary.

[0128] The monitoring area MA set in Figure 16 and the monitoring area set in Figure 8 are ultimately the same size, but if the size and / or number of products 70 displayed is different in the example of Figure 16, the size of the monitoring area MA set will also differ from that in the example of Figure 8.

[0129] For example, in the example of Figure 16, if the size of the product 70 is smaller than that in the example of Figure 8, or if the number of products 70 displayed is smaller than that in the example of Figure 8, the size of the monitoring area MA set in the example of Figure 16 will be smaller than that in the example of Figure 8.

[0130] It should be noted that when setting the monitoring area MA, it is not necessary to have all of the information on the width, height, and number of displayed products 70. When setting the monitoring area MA, the distance between the shelf labels 51 is detected by video recognition as described above, so even if some of the information on the width, height, and number of displayed products 70 is missing, the monitoring area setting unit 202 can supplement it with the detected distance. Furthermore, the width and / or height of the product 70 may be supplemented based on the size of the shelf label 51 detected by video recognition. The size of the shelf label 51 may be known, or may be detected by video recognition of the shelf label 51.

[0131] Furthermore, even if no shelf label 51 is installed as a reference for determining the distance between shelf labels 51 in the y-axis direction, such as on the top shelf of a display shelf, the monitoring area setting unit 212 can appropriately set the size of the monitoring area MA in the y-axis direction based on information indicating the height of the product 70.

[0132] Furthermore, since the number of monitored products 70 in the monitoring area MA can be ascertained, for example, the accuracy of gradual detection of shortages can be improved in the shortage detection unit 213. An example of gradual shortage detection will be described later.

[0133] (Example of correcting shelf tag detection results using shelf allocation information) Next, an example of correcting the shelf label detection result using the shelf allocation information 400 will be described when the shelf allocation information 400 includes, for example, information indicating the type of product 70 and information indicating the number of products 70 displayed.

[0134] For example, if the shelf allocation information 400 includes information such as "Three types of products 70 with different widths (X) are displayed on the m+1th shelf 52," the detected shelf label correction unit 216 (see FIG. 5) can use that information to detect and correct missed detection of shelf labels 51 by video recognition.

[0135] Fig. 17 shows a case where a shelf label 51 may go undetected. In the example of Fig. 17, three shelf labels 51 should be detected for the m+1 (=2)th shelf 52. Note that the shelf allocation information 400 includes information that indicates that products 70 of different widths (X1, X2, and X3) are displayed on each of the three shelf labels 51, which are represented by C1, C2, and C3 in Fig. 18.

[0136] However, because shelf label 51 at number 2 on the second row is partially overlapped by another label (obstacle) reading, for example, "SALE!", shelf label 51 at number 2 on the second row may fail to be detected by image recognition.

[0137] If the detection fails, the detected shelf label correction unit 216 can infer that a shelf label 51 is present at the second position on the second level by applying the width (X1, X2, and X3) and the number of displayed products for each product 70 to the m+1 (=2)th shelf board 52, for example, as shown in Fig. 19. The detected shelf label correction unit 216 may correct the missed detection by including the inferred shelf label 51 in the shelf labels 51 detected by video recognition.

[0138] In this way, even if there is a shelf tag 51 that fails to be detected by video recognition due to an obstacle, the missed detection by image recognition can be corrected based on the shelf allocation information 400, thereby improving the accuracy of setting the monitoring area MA based on the position of the shelf tag 51.

[0139] (PTZ camera application) For example, a PTZ camera may be used as the camera 10, and the shooting direction of the PTZ camera 10 may be changed and controlled, so that a plurality of display shelves 50 or a plurality of different display spaces of one display shelf 50 can be included in the monitoring targets.

[0140] Fig. 20 shows an application example of the PTZ camera 10. Fig. 20 shows an example in which two PTZ cameras 10 are installed, for example, on the ceiling of one of two display shelves 50 adjacent to each other in the depth direction.

[0141] Therefore, each of the PTZ cameras 10 captures, for example, the front of the other of the two display shelves 50 from diagonally above. Note that the PTZ camera 10 may be installed, for example, on the ceiling of the store in which the display shelves 50 are installed, and capture the front of the display shelves 50 from diagonally above.

[0142] For example, the shooting direction of each of the PTZ cameras 10 may be changed to a plurality of directions (e.g., N=3 directions) in the width direction of the display shelf 50. Here, "N" is an integer equal to or greater than 2 and represents the number of shooting directions. The change in the shooting direction may be controlled, for example, by the inventory monitoring device 20 communicating with the PTZ camera 10 via the communication IF 261 (see FIG. 5).

[0143] The change control of the shooting direction may be performed periodically at a preset cycle or irregularly at a specific timing. The shooting direction may also be changed by a user's instruction. For example, when a time period during which a specific product 70 is likely to run out of stock can be known in advance within a predetermined unit period such as one day, one week, or one month, the shooting direction may be controlled so that at least the product 70 that is likely to run out of stock during that time period is included in the monitoring target. A non-limiting example of a time period during which a specific product 70 is likely to run out of stock is a promotion time period (sale time period) that is held periodically or irregularly for the specific product 70.

[0144] Information indicating the shooting direction of the PTZ camera 10 (e.g., angle information) may be associated with the above-mentioned shelf allocation information 400. For example, an ID may be assigned to the PTZ camera 10 and N pieces of angle information (N is an integer equal to or greater than 2), and the ID may be associated with an ID in the shelf allocation information 400 (see FIG. 4).

[0145] By associating the IDs in this way, it is possible to identify, for example, which shelf level of which display shelf 50 is monitored by which shooting direction of which PTZ camera 10. The ID association may be performed by, for example, the shelf label association unit 215 (see FIG. 5).

[0146] Furthermore, as described above, the recognition model suitable for video recognition varies depending on the shooting direction, so a recognition model may be prepared in advance for each different shooting direction.

[0147] Furthermore, the recognition model suitable for video recognition may change depending on whether or not, for example, a glass door is attached to the front of the display shelf 50. Therefore, for example, a recognition model suitable for video recognition may be prepared in advance depending on the shooting direction and / or the presence or absence of a glass door.

[0148] 21 shows an example in which recognition models #1, #2, and #3 are set for three shooting directions #A, #B, and #C, respectively. Alternatively, for example, a plurality of recognition models may be switched to select a recognition model in which the number of detected shelf tags 51 best matches the shelf allocation information.

[0149] As described above, by using the PTZ camera 10 as the camera 10, it is possible to monitor and grasp inventory at multiple locations with one PTZ camera 10, which reduces the total number of cameras installed in the inventory management system 1. Therefore, the total cost of the inventory management system 1 can be reduced.

[0150] (Step-by-step stockout detection) Even if out-of-stock information is output after a shortage area of ​​the product 70 is detected, there are cases in which the product 70 cannot be replenished in time.

[0151] Therefore, for example, the inventory monitoring device 20 may present the inventory manager with the inventory level of the product 70 (which may be rephrased as "out-of-stock level") in stages according to the surface area ratio of the background image appearing in the monitoring area set by the monitoring area setting unit 212. Figures 22A to 22C show an example of the staged out-of-stock level.

[0152] Fig. 22A shows a situation where three products 70 are displayed from "row m, nth" to "row m, n+2th" (out-of-stock level 0). Fig. 22B shows a situation where one of the three products 70 is out of stock (out-of-stock level 1), and Fig. 22C shows a situation where three products 70 are out of stock (out-of-stock level 3).

[0153] The inventory monitoring device 20 (e.g., the stock-out detection unit 213) determines the stock-out level for the area ratio of the background image appearing in the monitoring area by, for example, setting the stock-out level in association with the area ratio of the background image appearing in the monitoring area at the stock-out level. Information indicating the determined stock-out level is output to, for example, the output unit 214.

[0154] The output unit 214 generates notification information including information indicating the stock-out level, and outputs it to the output device 203 and / or the communication unit 206. In this way, information indicating the gradual stock-out level is presented to, for example, an inventory manager.

[0155] Incidentally, the level at which the notification is issued among the tiered stockout levels (in other words, the threshold level for notification) may be set to a predetermined level in advance, or may be adaptively set by, for example, an inventory manager. In the example of Figs. 22A to 22C, for example, "stockout level 1" may be set as the threshold level for notification. Incidentally, the setting may be such that notification is issued at all stockout levels.

[0156] In this way, when multiple products 70 are displayed in a monitoring area, the inventory monitoring device 20 can present the inventory manager with gradual stockout levels. Therefore, the inventory manager can be alerted before the product 70 is completely out of stock, enabling timely product replenishment.

[0157] When the camera 10 is installed in a position where it photographs the shelf from diagonally above, the inventory monitoring device 20 may present the stockout level in the depth direction (z-axis direction) of the shelf to the inventory manager, as shown in Figures 23 and 24. For example, it may present the fact that only the product 70 in the front row is out of stock.

[0158] When the shelf allocation information 400 includes information indicating the number of products 70 displayed from the front of the shelf to the depth direction, the stockout level in the depth direction of the shelf may be set, for example, by the stockout detection unit 213 based on the video recognition results of the monitored area and the shelf allocation information 400.

[0159] (Moving object removal processing) When a person (e.g., a customer) passes in front of the display shelf 50 or stops temporarily in front of the display shelf 50, part or all of the shelf label 51 may be obscured by the person in the camera image, as shown in FIG. 25A, for example.

[0160] In such a case, a person captured in a camera image becomes a temporary obstacle to shelf label detection and missing item detection by image recognition, and the presence of a temporary obstacle can reduce the accuracy of shelf label detection and missing item detection.

[0161] Therefore, the inventory monitoring device 20 (for example, the shelf label detection unit 211 and / or the missing item detection unit 213) may remove moving objects such as people from the camera image as shown in FIG. 25B, as an example of pre-processing for image recognition.

[0162] If the camera 10 supports the moving object removal mode, the inventory monitoring device 20 may set the moving object removal mode to ON by communicating with the camera 10 via, for example, the communication IF 261 (see FIG. 5). Note that the specific procedure for removing moving objects is known from the above-mentioned Non-Patent Document 2 and the like, so a description thereof will be omitted.

[0163] By performing a moving object removal process on several frames of camera footage that contain moving objects, the accuracy of shelf label detection and missing item detection through video recognition can be improved.

[0164] (Effects of the embodiment) As described above, according to the above-mentioned embodiment, a surveillance area is set for the display shelf 50 in the camera image based on the position of the shelf label 51 detected by image recognition of the camera image including the display shelf 50, so that a surveillance area can be appropriately set for the camera image including the display shelf 50 without relying on manual operation, for example.

[0165] In other words, the setting of the display space to be monitored in the display shelf 50 can be automated. Therefore, for example, even if the relative positional relationship between the display shelf 50 and the camera 10 changes due to external factors such as vibration, it is possible to reset the appropriate monitoring area by detecting the shelf label 51 through video recognition. Note that the effect of the above-mentioned embodiment is not limited to automating the setting of the display space to be monitored entirely. Precision can also be improved by manualizing part of the setting, such as when the reference shelf label is manually specified.

[0166] The inventory monitoring device 20 monitors the inventory of the products 70 on the display shelves 50 based on changes in the image corresponding to the presence or absence of the products 70 in the set monitoring area, and notifies the monitoring results to, for example, an inventory manager, thereby preventing cases where a specific product is left out of stock. This prevents lost sales opportunities and improves the image of the product among customers.

[0167] Furthermore, by using the shelf allocation information 400, the inventory monitoring device 20 can notify, for example, an inventory manager of product information on each product 70 related to the detection of a shortage, so that the inventory manager can accurately grasp, for example, which product 70 is missing on which shelf. Therefore, even if it is difficult to identify the names of individual products by analyzing the camera images, the inventory manager can accurately and smoothly replenish the products 70 on each shelf, for example.

[0168] Furthermore, even if there is a shelf tag 51 that fails to be detected by video recognition due to an obstacle, the inventory monitoring device 20 can correct the missed detection by image recognition based on the shelf allocation information 400, thereby improving the accuracy of setting the monitoring area MA based on the position of the shelf tag 51.

[0169] In addition, the inventory monitoring device 20 can improve the accuracy of identifying, for example, the space or area occupied by multiple products 70 on a shelf level based on the shelf allocation information 400, thereby improving the accuracy of setting the monitoring area based on the shelf label position, and as a result, improving the accuracy of detecting when a product 70 is out of stock.

[0170] Furthermore, by using, for example, a PTZ camera as the camera 10, it is not necessary to install a camera 10 for each display shelf 50 or for each different area or space of the display shelf 50, so the number of cameras 10 installed in the inventory management system 1 can be reduced.

[0171] In this embodiment, a PTZ camera has been described as an example of camera 10, but a camera without a zoom function can be used as long as the shooting direction can be changed. Also, if there are restrictions on the arrangement direction of display shelf 50, camera 10 does not need to have both pan and tilt functions.

[0172] For example, if the display shelves 50 are arranged only in a substantially horizontal direction, even a camera 10 that only has a tilt function can capture images of a plurality of display shelves 50. In addition, the camera 10 may be an omnidirectional camera.

[0173] When an omnidirectional camera is used as the camera 10, the captured image is an omnidirectional image including images of a plurality of display shelves 50 existing around the camera 10. In this case, the control of changing the shooting direction corresponds to the process of changing and controlling the portion to be cut out from the captured omnidirectional image.

[0174] The inventory monitoring device 20 can achieve control similar to that achieved when a PTZ camera is used by performing shelf label detection and / or out-of-stock detection using a recognition model suitable for each virtual shooting direction corresponding to a portion cut out from the omnidirectional image.

[0175] In addition, since an omnidirectional image can include images of multiple display shelves 50, multiple display shelves 50 can be monitored simultaneously by simultaneously cutting out portions corresponding to each of the multiple display shelves 50 and performing shelf label detection and / or missing item detection using a recognition model appropriate for each.

[0176] In addition, the inventory monitoring device 20 can improve the accuracy of shelf label detection and / or missing item detection by using a recognition model suitable for image recognition, for example, depending on the shooting direction of the display shelf 50 and / or the presence or absence of a glass door.

[0177] Furthermore, when multiple products 70 are displayed in a monitoring area, the inventory monitoring device 20 can present the inventory manager with gradual stockout levels. This allows the inventory manager to be alerted before the product 70 runs out completely, enabling timely product replenishment.

[0178] Furthermore, by performing a moving object removal process on several frames of camera images that contain moving objects, the accuracy of shelf label detection and out-of-stock detection by image recognition in the inventory monitoring device 20 can be improved.

[0179] (others) In the above-described embodiment, an example has been described in which the object of monitoring by the computer 20 is the product 70 displayed on the display shelf 50, but the object of monitoring by the computer 20 is not limited to the product 70 related to commercial transactions. For example, the object of monitoring by the computer 20 may be an "item" such as an exhibit displayed in a showcase. In this case, for example, the computer 20 can detect the loss of an "item" in the showcase and notify a manager or the like.

[0180] The computer 20 that monitors the inventory or display status of "items" may be called, for example, a "shelf monitoring device," an "item monitoring device," or an "item display status monitoring device." Also, a program that causes the computer 20 to function as a device that monitors the inventory or display status of "items" may be called a "shelf monitoring program," an "item monitoring program," or an "item display status monitoring program."

[0181] Furthermore, each functional block used in the description of the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip that includes some or all of the functional blocks. Here, we refer to it as an LSI, but depending on the level of integration, it may also be called an IC, a system LSI, a super LSI, or an ultra LSI.

[0182] The method of integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. After LSI manufacturing, a field programmable gate array (FPGA) that can be programmed, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used.

[0183] Furthermore, if an integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a different derived technology, it is natural that such technology may be used to integrate the functional blocks. The application of biotechnology, etc. is also a possibility.

[0184] Summary of the Disclosure The shelf monitoring device of the present disclosure may include a detection unit that detects shelf tags attached to the display shelves corresponding to items displayed on the display shelves by video recognition of video data including the display shelves, a setting unit that sets a monitoring area in which the items are to be displayed on the display shelves in the video data based on the position of the detected shelf tag, a monitoring unit that monitors the display status of the items on the display shelves based on image changes corresponding to the presence or absence of the items in the monitoring area, and an output unit that outputs the monitoring results by the monitoring unit.

[0185] In addition, the shelf monitoring device according to the present disclosure may include a communication unit that receives shelf allocation information including information about items to be displayed in each of the monitoring areas, and an association unit that associates information about the items included in the shelf allocation information with the shelf label detected by the detection unit. When the monitoring result of the monitoring area indicates a shortage or a shortage of the item, the output unit may output the monitoring result and information about the item associated with the shelf label corresponding to the monitoring area.

[0186] In addition, the shelf monitoring device of the present disclosure may include a communication unit that receives shelf allocation information including at least one of the size and the number of items to be displayed in each of the monitoring areas, and a correction unit that corrects the position of the shelf tag detected by the detection unit based on at least one of the size and the number of items included in the shelf allocation information.

[0187] In addition, the shelf monitoring device of the present disclosure may be equipped with a communication unit that receives shelf allocation information including at least one of the size and number of items to be displayed in each of the monitoring areas, and the setting unit may determine the size of the monitoring area based on at least one of the size and number of items included in the shelf allocation information.

[0188] In the shelf monitoring device according to the present disclosure, the video data may include a plurality of pieces of video data captured in a plurality of different shooting directions by at least a camera whose shooting direction is variable. The detection unit may detect the shelf label by video recognition using a recognition model corresponding to the shooting direction.

[0189] Furthermore, in the shelf monitoring device of the present disclosure, the monitoring unit may gradually detect the inventory level of the item based on the area proportion of the background image appearing in the monitored area, and the output unit may output information indicating the inventory level at a stage before the item runs out of stock.

[0190] In addition, the shelf monitoring device of the present disclosure may be equipped with a communication unit that receives shelf allocation information including information on the number of items to be displayed in each of the monitored areas, and the monitoring unit may gradually detect the inventory level of the items based on the area proportion of the background image appearing in the monitored area and the number of items displayed included in the shelf allocation information.

[0191] Furthermore, in the shelf monitoring device of the present disclosure, the setting unit may calculate, for each detected shelf label, the distance between the shelf label and an adjacent shelf label, and set a monitoring area corresponding to each of the shelf labels based on the distance.

[0192] Furthermore, in the shelf monitoring device of the present disclosure, the shelf labels may be arranged in accordance with a predetermined rule for each area in which the items are displayed, and the setting unit may set the monitoring area based on the distance between each of the shelf labels and the adjacent shelf labels and the predetermined rule.

[0193] Furthermore, in the shelf monitoring device of the present disclosure, for a shelf label that has no adjacent shelf label, the setting unit may set a monitoring area that corresponds to the shelf label that has no adjacent shelf label based on the distances calculated for the other shelf labels.

[0194] Furthermore, in the shelf monitoring method of the present disclosure, by performing image recognition on video data including a display shelf, shelf tags attached to the display shelf corresponding to items displayed on the display shelf are detected, a monitoring area in which the items are to be displayed for the display shelf in the video data is set based on the position of the detected shelf tag, the display status of the items on the display shelf is monitored based on changes in the image corresponding to the presence or absence of the items in the monitoring area, and the results of the monitoring are output.

[0195] In addition, the shelf monitoring program of the present disclosure causes a computer to execute processes to detect shelf tags attached to the display shelves corresponding to items displayed on the display shelves through image recognition of video data including the display shelves, set a monitoring area in which the items are to be displayed for the display shelves in the video data based on the position of the detected shelf tag, monitor the display status of the items on the display shelves based on image changes corresponding to the presence or absence of the items in the monitoring area, and output the results of the monitoring.

[0196] The disclosures of the specification, drawings and abstract contained in Japanese Patent Application No. 2017-209440, filed on October 30, 2017, are incorporated herein by reference in their entirety. [Industrial Applicability]

[0197] The present disclosure is suitable for a system for monitoring or managing the display status of items. [Explanation of symbols]

[0198] 1. Inventory management system 10. Camera 20 Computer (inventory monitoring device) 50 display shelves 51 Shelf Tag 52 Shelf 70 items 201 Processor 202 Input Device 203 Output Device 204 Memory 205 Storage 206 Communications Department 211 Shelf label detector 212 Monitoring area setting section 213 Missing Item Detection Unit 214 Output section 215 Shelf label linking section 216 Detector shelf tag correction unit 261,262 Communication Interface (IF) 400 Shelf information MA Surveillance Area

Claims

1. a setting unit that acquires video data capturing the display shelf from an obliquely upward direction and sets a monitoring area in which items are displayed on the display shelf; a detection unit that detects the inventory level of the items in stages based on an area ratio of the background image using a trained model for recognizing a bottom surface of the display shelf in the monitoring area as a background image, and detects the inventory level in a depth direction of the display shelf; an output unit that outputs information indicating an inventory level of the item and information indicating an inventory level in a depth direction of the display shelf; Equipped with the setting unit sets a monitoring area in which the items are to be displayed on the display shelf in the video data based on a position of a shelf tag attached to the display shelf corresponding to the item to be displayed on the display shelf; Shelf monitoring device.

2. the video data includes video data captured in a plurality of different shooting directions by at least a camera whose shooting direction is variable, The detection unit detects an area ratio of the background image using a trained model corresponding to the shooting direction. The shelf monitor of claim 1.

3. The video data is video data capturing a display shelf from the front, and the background video includes a back panel of the display shelf. The shelf monitoring device of claim 2.

4. The video data is video data capturing a display shelf from an oblique side, and the background video includes a partition board of a shelf of the display shelf. The shelf monitoring device of claim 2.

5. the output unit outputs information indicating the inventory level to a display, and the information indicating the inventory level is information obtained by changing a color of the out-of-stock area detected by the detection unit to a color that stands out more than other areas. The shelf monitor of claim 1.

6. the output unit outputs information indicating the inventory level to a display, and the information indicating the inventory level is information that displays the color of the out-of-stock area detected by the detection unit with a solid line frame or a dotted line frame. The shelf monitor of claim 1.

7. the detection unit determines an inventory level relative to the area ratio of the background image appearing in the monitoring area by associating an inventory level with an area ratio of the background image appearing in the monitoring area at the inventory level, and the output unit outputs the determined stock-out level as information indicating the stock-out level. The shelf monitor of claim 1.

8. The detection unit deletes moving objects from the video data in advance in the video recognition. The shelf monitor of claim 1.

9. a communication unit for receiving shelf allocation information including information on the number of items to be displayed in each of the monitoring areas, the detection unit detects the inventory level of the item in stages based on an area ratio of the background image appearing in the monitoring area and the number of items displayed that is included in the shelf allocation information; The shelf monitor of claim 1.

10. A computer acquires image data capturing the display shelf from an obliquely upward direction, and sets a monitoring area in which items are displayed on the display shelf; a computer detects an inventory level of the item in a stepwise manner based on an area ratio of the background image using a trained model for recognizing the bottom surface of the display shelf in the monitoring area as a background image, and detects an inventory level in a depth direction of the display shelf; a computer outputs information indicating an inventory level of the item and information indicating an inventory level in a depth direction of the display shelf; setting a monitoring area in which the items are to be displayed on the display shelf in the video data based on the positions of shelf tags attached to the display shelf corresponding to the items to be displayed on the display shelf; Shelf monitoring method.

11. acquiring image data capturing the display shelf from an obliquely upward direction, and setting a monitoring area in which items are displayed with respect to the display shelf; using a trained model for recognizing the bottom surface of the display shelf in the monitoring area as a background image, detecting the inventory level of the item in stages based on an area ratio of the background image, and detecting the inventory level in the depth direction of the display shelf; outputting information indicating the inventory level of the item and information indicating the inventory level in the depth direction of the display shelf; setting a monitoring area in which the items are to be displayed on the display shelf in the video data based on the positions of shelf tags attached to the display shelf corresponding to the items to be displayed on the display shelf; The process is executed by a computer. Shelf monitoring program.

Images