Commodity display state detection system, management apparatus, commodity display state detection method, and commodity display state detection program

Abstract

To provide a commodity display state detection system, a management apparatus, a commodity display state detection method, and a commodity display state detection program for smoothly and efficiently detecting a display state of commodities in a commodity shelf even when a store is crowded.SOLUTION: In a store, a camera installed to image a side position of a cart images a state of commodities displayed in a commodity shelf when a customer with the cart passes through commodity shelves of the commodities and transmits a captured video to a management apparatus. The management apparatus issues (S107), when detecting stockout or low stock of a displayed commodity (Yes in S103), using the received video (S101), a replenishment request pertaining to the commodity to a store clerk terminal 30.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a product display state detection system, a management device, a product display state detection method, and a product display state detection program that can smoothly and efficiently detect the display state of products on a product shelf even when a store is crowded.

Background Art

[0002] When operating a store, if a product displayed on a product shelf is out of stock, it is undesirable because customers will lose the opportunity to purchase the product. Frequent out-of-stock situations may lead customers to have the impression that the store has many out-of-stock items, which may also result in customer loss. Also, when the display state of products is messy, it may similarly lead to customer loss. For these reasons, it is necessary to monitor the products displayed on the product shelves. However, when store employees visually monitor, the work efficiency decreases.

[0003] For this reason, prior arts for monitoring out-of-stock items without imposing a burden on store employees are known. For example, Patent Document 1 discloses a system that captures an image of a product shelf with a camera movably provided on the ceiling, detects out-of-stock items based on the captured image, and transports out-of-stock products from the backyard with an autonomous robot.

[0004] Also, Patent Document 2 discloses a technique for automatically generating waypoints at which a robot captures an image of a shelf when an autonomous driving robot captures an image of a product shelf in a store and detects out-of-stock items or the like based on the captured image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when a camera is attached to the ceiling as in Patent Document 1 above, there is a problem that it is difficult to detect out-of-stock items on low shelves in terms of angle. Also, when a camera is attached to an autonomous mobile robot as in Patent Document 2, imaging becomes difficult because there is a possibility that the robot will collide with customers when the store is crowded with people. In addition, the introduction cost of the robot is also high.

[0007] Also, a plan to attach a camera to another product shelf facing a certain product shelf can be considered, but since there is a possibility that customers will enter between the camera and the product shelf, it may not be possible to image the product shelf when the store is crowded. To solve this problem, it is also possible to install many cameras, but the more cameras are added, the higher the introduction cost will be.

[0008] The present invention has been made to solve the above problems of the prior art, and an object thereof is to provide a product display state detection system, a management device, a product display state detection method, and a product display state detection program that can smoothly and efficiently detect the display state of products on product shelves even when the store is crowded.

Means for Solving the Problems

[0009] To solve the above problems, the present invention includes an imaging unit installed on a cart in which a customer stores products, a detection unit that detects the display state of products on a product shelf in the store from an image of the product shelf captured by the imaging unit, and a notification unit that notifies predetermined product information according to the display state of products on the product shelf detected by the detection unit. The imaging unit is installed so as to be able to image the side position of the cart with respect to the traveling direction of the cart.

[0010] Also, in the present invention, the notification unit notifies the product information when the display state of products on the product shelf is an out-of-stock state or a low-stock state.

[0011] Further, in the present invention, when the display state of the products on the product shelf is disordered, the notification unit notifies the product information.

[0012] Further, in the present invention, in the above invention, it further includes a first storage unit for storing the inventory information of the products in the store, and when the display state of the products on the product shelf is out of stock or in short supply, the notification unit refers to the inventory information of the products stored in the first storage unit, and when the inventory quantity of the out-of-stock or out-of-stock products is equal to or more than a predetermined threshold, it notifies the replenishment request for the product, and when the inventory quantity of the out-of-stock or out-of-stock products is less than the predetermined threshold, it does not notify the replenishment request for the product.

[0013] Further, in the present invention, in the above invention, it further includes a second storage unit for storing the shelf allocation information associating the identification information of the product shelf with the types of products displayed on the product shelf, and the detection unit detects the identification information of the product shelf from the image of the product shelf, and when the products on the product shelf are out of stock or in short supply, it refers to the shelf allocation information stored in the second storage unit to specify the types of products associated with the identification information of the product shelf.

[0014] Further, in the present invention, in the above invention, it further includes a third storage unit for storing a reference image of the product shelf captured in advance, and the detection unit detects a change in the display state as the display state of the products on the product shelf based on the image of the product shelf captured by the imaging unit and the reference image of the product shelf stored in the third storage unit.

[0015] Further, the present invention is a management device in a product display state detection system for detecting the display state of products on the product shelves in a store, which is installed on a cart for customers to store products, and includes a detection unit for detecting the display state of the products on the product shelf from an image of the product shelf captured by an imaging unit capable of imaging a lateral position of the cart with respect to the traveling direction of the cart, and a notification unit for notifying predetermined product information according to the display state of the products on the product shelf detected by the detection unit.

[0016] The present invention also relates to a method for detecting a product display state in a product display state detection system that detects the display state of products on a product shelf in a store. The method includes a detection step of detecting the display state of products on the product shelf from an image of the product shelf captured by an imaging unit installed on a cart in which a customer stores products and capable of imaging a lateral position of the cart with respect to the traveling direction of the cart, and a notification step of notifying predetermined product information according to the display state of products on the product shelf detected in the detection step.

[0017] The present invention also relates to a product display state detection program executed by a management device in a product display state detection system that detects the display state of products on a product shelf in a store. The program executes a detection procedure of detecting the display state of products on the product shelf from an image of the product shelf captured by an imaging unit installed on a cart in which a customer stores products and capable of imaging a lateral position of the cart with respect to the traveling direction of the cart, and a notification procedure of notifying predetermined product information according to the display state of products on the product shelf detected in the detection procedure.

Advantages of the Invention

[0018] According to the present invention, even when the store is crowded, the display state of products on the product shelf can be detected smoothly and efficiently.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

MODE FOR CARRYING OUT THE INVENTION

[0020] [Embodiment] Hereinafter, embodiments of a product display state detection system, a management device, a product display state detection method, and a product display state detection program will be described in detail with reference to the drawings.

[0021] <Overview of Product Display State Detection System According to Embodiment> First, an overview of the product display state detection system according to the present embodiment will be described. FIG. 1 is an explanatory diagram of an overview of the product display state detection system according to the present embodiment.

[0022] As shown in FIG. 1, in the product display state detection system, when a customer passes between product shelves in a store while pushing a cart 10, a camera 20 installed so as to be able to image the side position of the cart 10 images the display state of the products on the product shelves, and transmits the captured video to a management device 40 (S1).

[0023] If the management device 40 detects out-of-stock or shortage of the displayed products using the received video (S2), it notifies the store staff terminal 30 of a replenishment request for the corresponding products (S3).

[0024] As described above, in the product display state detection system according to this embodiment, the display state of the products on the product shelf is imaged by a camera installed so as to be able to image the side position of the cart used by the customer, and using the imaged video, out-of-stock or shortage of the products on the product shelf is detected. Therefore, even when the store is crowded, the display state of the products on the product shelf can be detected smoothly and efficiently.

[0025] <System Configuration of the Product Display State Detection System According to the Embodiment> Next, the system configuration of the product display state detection system according to this embodiment will be described. FIG. 2 is a diagram showing the system configuration of the product display state detection system according to this embodiment.

[0026] As shown in FIG. 2, the management device 40 installed in the store is communicably connected to the camera 20 and the store staff terminal 30 via a wireless router 50. The wireless router 50 is connected to the camera 20 and the store staff terminal 30 by short-range wireless communication such as Wi-Fi (registered trademark).

[0027] The camera 20 is an imaging device installed on the front surface of the cart 10 in order to be able to image the side position of the cart 10. The camera 20 has one lens on each of its both side surfaces, images both side positions simultaneously, and transmits the imaged video to the management device 40 via the wireless router 50.

[0028] The store staff terminal 30 is a terminal such as a tablet carried by the store staff. If the store staff terminal 30 receives a replenishment request from the management device 40, it displays the received replenishment request. If the store staff terminal 30 receives a disorder confirmation request from the management device 40, which is a request to confirm whether the display state of the products on the product shelf is disordered, it displays the received disorder confirmation request.

[0029] The management device 40 is a device that detects the display state of products on the product shelves. When the management device 40 receives data related to products, inventory thresholds, inventory quantities, or display information of products for each product shelf, it stores the received data in product management data, inventory threshold data, inventory data, or product shelf data.

[0030] Also, when the management device 40 receives an image from the camera 20, it extracts an image in which the entire area of each product shelf is photographed, and identifies the product shelf number from the extracted image. Then, it determines the display state of the products on the product shelf corresponding to the extracted image.

[0031] Also, if the management device 40 determines that there is a shortage or out-of-stock in the determination of the display state of the products on the product shelf, it identifies the product number corresponding to the determined shelf section and notifies the store clerk terminal 30 as a replenishment request.

[0032] Also, if the management device 40 determines that there is disorder in the determination of the display state of the products on the product shelf, it identifies the product number corresponding to the determined shelf section and notifies the store clerk terminal 30 as a disorder confirmation request.

[0033] <Configuration of the management device 40> Next, the configuration of the management device 40 shown in FIG. 2 will be described. FIG. 3 is a functional block diagram showing the configuration of the management device 40 shown in FIG. 2. As shown in FIG. 3, the management device 40 is connected to a display unit 41 and an input unit 42, and has a communication unit 44, a storage unit 45, and a control unit 46.

[0034] The display unit 41 is a display device such as a liquid crystal panel or a display device. The input unit 42 is an input device such as a keyboard or a mouse. The communication unit 44 is an interface unit for data communication with the camera 20 and the store clerk terminal 30 via a wireless router 50.

[0035] The storage unit 45 is a storage device such as a hard disk device or a non-volatile memory, and stores product management data 45a, inventory threshold data 45b, inventory data 45c, product shelf data 45d, and detection data 45e.

[0036] The merchandise management data 45a is data indicating information about the merchandise. The inventory threshold data 45b is data indicating the inventory threshold used for determining replenishment requests for the merchandise. The inventory data 45c is data indicating the inventory quantity of the merchandise stored in the backyard.

[0037] The merchandise shelf data 45d is data indicating the merchandise numbers displayed on the merchandise shelves and indicating shelf division information. The detection data 45e is data indicating the images extracted for each merchandise shelf from the video captured by the camera 20.

[0038] The control unit 46 is a control unit that performs overall control of the management device 40, and includes a merchandise management unit 46a, an inventory threshold setting unit 46b, an inventory management unit 46c, a merchandise shelf management unit 46d, a detection unit 46e, a status determination unit 46f, a merchandise identification unit 46g, an inventory determination unit 46h, and a notification unit 46i. Actually, by loading these programs into the CPU (Central Processing Unit) and executing them, processes corresponding to the merchandise management unit 46a, the inventory threshold setting unit 46b, the inventory management unit 46c, the merchandise shelf management unit 46d, the detection unit 46e, the status determination unit 46f, the merchandise identification unit 46g, the inventory determination unit 46h, and the notification unit 46i are executed respectively.

[0039] The merchandise management unit 46a is a processing unit that manages the merchandise management data 45a. If the merchandise management unit 46a receives data related to the merchandise from the input unit 42, it stores the received data in the merchandise management data 45a.

[0040] The inventory threshold setting unit 46b is a processing unit that manages the inventory threshold data 45b. If the inventory threshold setting unit 46b receives an inventory threshold related to the merchandise from the input unit 42, it stores the received inventory threshold in the inventory threshold data 45b.

[0041] The inventory management unit 46c is a processing unit that manages the inventory data 45c. If the inventory management unit 46c receives an inventory quantity related to the merchandise from the input unit 42, it stores the received inventory quantity in the inventory data 45c.

[0042] The merchandise shelf management unit 46d is a processing unit that manages merchandise shelf data 45d. When the merchandise shelf management unit 46d receives the display information of the merchandise for each merchandise shelf from the input unit 42, it stores the received display information of the merchandise for each merchandise shelf in the merchandise shelf data 45d. Note that the display information of the merchandise for each merchandise shelf includes the merchandise numbers for each of the upper, lower, left, and right shelf sections of the merchandise shelf. For example, the upper and lower shelf sections are divided into an upper section, a middle section, and a lower section, and the left and right shelf sections are divided into a left shelf, a middle shelf, and a right shelf.

[0043] The detection unit 46e is a processing unit that manages detection data 45e. When the detection unit 46e receives an image from the camera 20, it extracts an image in which the entire area of each merchandise shelf is photographed, and identifies the merchandise shelf number from the extracted image. Then, it associates the extracted image with the merchandise shelf number and stores them in the detection data 45e.

[0044] The status determination unit 46f is a processing unit that determines the display status of the merchandise on the merchandise shelf. When the detection data 45e is updated by the detection unit 46e, the status determination unit 46f determines the display status of the merchandise on the merchandise shelf related to the updated data. Specifically, it extracts the area of the shelf section from the image of the updated detection data 45e, and determines the presence or absence of out-of-stock, low stock, or disorder for each shelf section.

[0045] For example, if all parts of the back of the shelf are exposed within the extracted area, it is determined as out-of-stock. If more than half of the back of the shelf is exposed within the extracted area, it is determined as low stock. Also, if one side of the exposed part of the back of the shelf within the extracted area is formed by diagonal lines, it is determined as disorder. The formation of these diagonal lines indicates a situation where the merchandise has fallen.

[0046] The merchandise identification unit 46g is a processing unit that identifies the merchandise numbers determined as out-of-stock, etc. by the status determination unit 46f. When the merchandise identification unit 46g determines that it is out-of-stock or low stock by the status determination unit 46f, it identifies the merchandise numbers corresponding to the determined shelf sections from the merchandise shelf data 45d, and transfers the identified merchandise numbers and merchandise shelf numbers to the inventory determination unit 46h.

[0047] Further, if the product identification unit 46g determines that there is disorder based on the determination by the state determination unit 46f, it identifies the product number corresponding to the determined shelf section from the product shelf data 45d, and passes on a disorder confirmation instruction including the identified product number and product shelf number to the notification unit 46i.

[0048] The inventory determination unit 46h is a processing unit that determines the inventory of the product related to the product number and product shelf number specified by the product identification unit 46g. When the inventory determination unit 46h receives the product number and product shelf number from the product identification unit 46g, it identifies the inventory quantity related to this product number from the inventory data 45c, and identifies the inventory threshold from the inventory threshold data 45b. If the identified inventory quantity is equal to or greater than the inventory threshold, it passes on a replenishment instruction including the product number and product shelf number to the notification unit 46i.

[0049] The notification unit 46i is a processing unit that notifies requests for product replenishment and disorder confirmation. When the notification unit 46i receives a replenishment instruction from the inventory determination unit 46h, it notifies this replenishment instruction to the store clerk terminal 30 as a replenishment request. When the notification unit 46i receives a disorder confirmation instruction from the product identification unit 46g, it notifies this disorder confirmation instruction to the store clerk terminal 30 as a disorder confirmation request.

[0050] Next, an example of the data stored in the storage unit 45 of the management device 40 shown in FIG. 3 will be described. FIGS. 4 and 5 are diagrams showing an example of the product management data 45a, inventory threshold data 45b, inventory data 45c, product shelf data 45d, and detection data 45e shown in FIG. 3.

[0051] The product management data 45a shown in FIG. 4(a) associates the product number "11001" with the product category "daily necessities" and the product name "Power Detergent Cleaner", and associates the product number "11002" with the product category "daily necessities" and the product name "Foaming Detergent Fluffy".

[0052] The inventory threshold data 45b shown in FIG. 4(b) associates the inventory threshold "5" with the product number "11001" and associates the inventory threshold "10" with the product number "11002".

[0053] The inventory data 45c shown in FIG. 4(c) associates the inventory quantity "20" with the product number "11001" and the inventory quantity "7" with the product number "11002".

[0054] The product shelf data 45d shown in FIG. 5(a) associates the product number "11001" on the left shelf, the product number "11002" in the middle shelf, and the product number "11003" on the right shelf with the product shelf number "2101" and the upper / lower shelf classification "upper", and associates the product number "11013" on the left shelf, the product number "11015" in the middle shelf, and the product number "11019" on the right shelf with the product shelf number "2101" and the upper / lower shelf classification "middle".

[0055] In addition, the product shelf data 45d associates the product number "11022" on the left shelf, the product number "11027" in the middle shelf, and the product number "11033" on the right shelf with the product shelf number "2101" and the upper / lower shelf classification "lower", and associates the product number "12005" on the left shelf, the product number "12011" in the middle shelf, and the product number "12026" on the right shelf with the product shelf number "2102" and the upper / lower shelf classification "upper".

[0056] The detection data 45e shown in FIG. 5(b) associates the detection data "2101_01.jpg" with the product shelf number "2101" and the detection data "2102_01.jpg" with the product shelf number "2102".

[0057] <An example of determining the product display state according to the embodiment> Next, an example of determining the product display state according to the present embodiment will be described. FIGS. 6 and 7 are diagrams showing an example of determining the product display state according to the present embodiment.

[0058] As shown in FIG. 6, an image in which the entire product shelf is photographed is extracted from the video captured by the camera 20, and this extracted image is stored as detection data in association with the product shelf number. For example, in the product shelf with the product shelf number "2101", products are displayed in a total of 9 shelf sections divided by the upper, middle, and lower sections, as well as the left, middle, and right shelves.

[0059] As shown in FIG. 7, the area of the shelf section is extracted from the extracted data, and the product display state is determined. Specifically, as shown in FIG. 7(a), if more than half of the back part of the shelf is exposed within the extracted area, it is determined as out of stock. As shown in FIG. 7(b), if the entire back part of the shelf is exposed within the extracted area, it is determined as out of stock. As shown in FIG. 7(c), if one side of the exposed back part of the shelf within the extracted area is formed by diagonal lines, it is determined as disordered.

[0060] <Processing procedure of the management device 40 according to the embodiment> Next, the processing procedure of the management device 40 according to the present embodiment will be described. FIG. 8 is a flowchart showing the processing procedure of the management device 40 according to the present embodiment.

[0061] As shown in FIG. 8, if the management device 40 has not received an image from the camera 20 (step S101: No), it returns to step S101. If it has received an image from the camera 20 (step S101: Yes), it extracts an image in which the entire product shelf is photographed from the received image. And if the shelf number has not been specified (step S102: No), it returns to step S102. If the shelf number has been specified (step S102: Yes), it determines the product display state.

[0062] If it is determined as out of stock or out of stock by the determination of the product display state (step S103: Yes), it specifies the product number corresponding to the determined shelf section from the product shelf data 45d. If the stock quantity related to the specified product number is equal to or more than the stock threshold (step S104: Yes), it notifies the replenishment request to the clerk terminal 30 (step S105). Note that in step S103, when it is not determined as out of stock or out of stock (step S103: No), and in step S104, when the stock quantity is less than the stock threshold (step S104: No), it proceeds to step S106.

[0063] If it is determined that there is no disorder based on the determination of the product display state (step S106: No), the process ends as it is. If it is determined that there is disorder (step S106: Yes), the product number corresponding to the determined shelf section is specified from the product shelf data 45d. Then, a disorder confirmation request including the specified product number and product shelf number is notified to the store clerk terminal 30 (step S107), and the process ends.

[0064] As described above, in the product display state detection system according to the present embodiment, the display state of the products on the product shelf is imaged by a camera installed so as to be able to image the side position of the cart used by the customer, and using the imaged video, out-of-stock, shortage, or disorder of the products on the product shelf is detected. Therefore, even when the store is crowded, the display state of the products on the product shelf can be detected smoothly and efficiently.

[0065] Note that in the above-described Embodiment 1, the configuration for detecting out-of-stock, shortage, or disorder of the products on the product shelf and notifying the result to the store clerk terminal has been described, but the present invention is not limited to this. It is also possible to configure to detect out-of-stock, shortage, or disorder of the products on the product shelf and notify and display the result on a display such as a PC or digital signage installed in the store.

[0066] Also, in the above-described Embodiment 1, the configuration for detecting out-of-stock and shortage of the products on the product shelf and notifying the result to the store clerk terminal has been described, but the present invention is not limited to this. It is also possible to configure to detect out-of-stock and shortage of the products on the product shelf and notify the replenishment quantity to the store clerk terminal when replenishment is necessary.

[0067] Also, in the above-described Embodiment 1, the configuration for imaging the display state of the products on the product shelf by a camera installed so as to be able to image the side position of the cart used by the customer has been described, but the present invention is not limited to this. The number of carts on which the camera is installed may be all the carts in the store or some of the carts. Also, the mounting position of the camera may be any part of the cart as long as it can image the side product shelf.

[0068] In the above-described Embodiment 1, the configuration of identifying the product shelf imaged by the camera using the product shelf number has been described. However, the present invention is not limited thereto. In order to identify the product shelf, it is also possible to configure to notify the image of the entire product shelf captured by the camera to the terminal for store clerks. This is because the store clerk can determine the position of this product shelf by checking the image of the entire product shelf captured.

[0069] In the above-described Embodiment 1, the configuration of specifying the product number from the shelf section of the product shelf imaged by the camera has been described. However, the present invention is not limited thereto. It is also possible to configure to specify the product number from the product tags attached to each shelf of the product shelf imaged by the camera.

[0070] In the above-described Embodiment 1, the configuration of installing a camera having two lenses on the front surface of the cart in order to be able to image the side position of the cart has been described. However, the present invention is not limited thereto. It is also possible to configure to install one or a plurality of cameras on the side surface of the cart so as to be able to image the side position of the cart. Note that the imaging direction may be such that only one side direction of the cart can be imaged. Also, an imaging function may be added to the operation terminal of the smart cart so that both sides or one side of the cart can be imaged.

[0071] [Modification Example 1] By the way, in the above-described Embodiment 1, the configuration of imaging the display state of the products on the product shelf by the camera installed so as to be able to image the side position of the cart used by the customer and detecting the out-of-stock, shortage or disorder of the products on the product shelf using the imaged video has been described. However, the present invention is not limited thereto.

[0072] A customer shopping in the store may shop with a shopping basket without using a cart. Even in this case, it is also possible to configure to image the display state of the products on the product shelf. Specifically, it is also possible to configure to image the display state of the products on the product shelf by installing cameras in front of and behind the shopping basket.

[0073] In the first modification example, a product display state detection system that captures the display state of products on a product shelf by installing cameras in front of and behind a shopping basket will be described.

[0074] <Overview of the product display state detection system according to Modification Example 1> The overview of the product display state detection system according to the first modification example will be described. FIG. 9 is an explanatory diagram of the overview of the product display state detection system according to the first modification example.

[0075] As shown in FIG. 9, in the product display state detection system according to the first modification example, cameras 20a and 20b are installed in front of and behind a shopping basket 90. When a customer with the shopping basket 90 faces the product shelf, the camera 20 can capture the product shelf. Using the captured video, the display state of the products on the product shelf can be determined.

[0076] As described above, in the product display state detection system according to the first modification example, since cameras are installed in front of and behind the shopping basket to capture the display state of the products on the product shelf, even when the store is crowded, the display state of the products on the product shelf can be detected smoothly and efficiently.

[0077] [Modification Example 2] By the way, in the above-described Embodiment 1, the display state of the products on the product shelf is captured by a camera installed so as to be able to capture the side position of the cart used by the customer, and the area of the shelf section is extracted from the captured video, and the out-of-stock, shortage, or disorder of the products for each shelf section is detected. However, the present invention is not limited to this.

[0078] It is also possible to configure to capture each product shelf at the time of opening and compare it with the image captured by the camera of the cart during business hours to detect the out-of-stock, shortage, or disorder of the products on each product shelf.

[0079] In this second modification example, a product display state detection system will be described that captures images of each product shelf at the time of store opening and compares them with the images captured by the camera on the cart during business hours to detect out-of-stock items, shortages, or disarray of products on each product shelf.

[0080] <Configuration of the management device 100> The configuration of the management device 100 according to this second modification example will be described. FIG. 10 is a functional block diagram showing the configuration of the management device 100 according to this second modification example.

[0081] As shown in FIG. 10, the management device 100 is connected to a display unit 41 and an input unit 42, and includes a communication unit 44, a storage unit 110, and a control unit 120. Note that the description of the functional units similar to those of the management device 40 shown in FIG. 3 will be omitted.

[0082] The storage unit 110 is a storage device such as a hard disk drive or a non-volatile memory, and stores product management data 45a, stock threshold data 45b, stock data 45c, product shelf data 45d, initial display state data 111, and detection data 45e. The initial display state data 111 is data indicating images of each product shelf at the time of store opening.

[0083] The control unit 120 is a control unit that performs overall control of the management device 100, and includes a product management unit 46a, a stock threshold setting unit 46b, a stock management unit 46c, a product shelf management unit 46d, an initial display state acquisition unit 121, a detection unit 46e, a state determination unit 122, a product identification unit 46g, a stock determination unit 46h, and a notification unit 46i. Actually, by loading these programs into the CPU and executing them, processes corresponding to the product management unit 46a, the stock threshold setting unit 46b, the stock management unit 46c, the product shelf management unit 46d, the initial display state acquisition unit 121, the detection unit 46e, the state determination unit 122, the product identification unit 46g, the stock determination unit 46h, and the notification unit 46i will be executed respectively.

[0084] The initial display state acquisition unit 121 is a processing unit that manages the initial display state data 111. During store opening preparation, if the initial display state acquisition unit 121 receives video from the camera 20, it extracts an image in which the entire area of each product shelf is captured, associates the extracted image with the product shelf number, and stores it in the initial display state data 111.

[0085] The state determination unit 122 is a processing unit that determines the display state of products on the product shelves. When the detection data 45e is updated by the detection unit 46e, the state determination unit 122 identifies the product shelf number from the image of the updated data, extracts the initial display state image corresponding to this product shelf number from the initial display state data 111, and then compares the image of the updated data with the initial display state image to determine the display state of the products on the product shelf.

[0086] Next, an example of the data stored in the storage unit 110 of the management device 100 shown in FIG. 10 will be described. FIG. 11 is a diagram showing an example of the initial display state data 111 shown in FIG. 10.

[0087] The initial display state data 111 shown in FIG. 11 associates the state where the initial display state image is "2101_00.jpg" with the product shelf number "2101", and associates the state where the initial display state image is "2102_00.jpg" with the product shelf number "2102".

[0088] <An example of determining the product display state according to Modification 2> Next, an example of determining the product display state according to this Modification 2 will be described. FIG. 12 is a diagram showing an example of determining the product display state according to this Modification 2.

[0089] As shown in FIG. 12, the image of the product shelf extracted using the video received from the camera 20 during business hours shows out-of-stock, low inventory, or disorder of the products. To determine this, it is compared with the initial display state image at the time of store opening.

[0090] Specifically, if Figure 6 is regarded as the initial display state image, the upper right shelf is determined to be out of stock, the upper left shelf and the middle right shelf are determined to be out of stock, and the middle lower shelf is determined to be in a mess. For example, for each of the nine shelf sections of the product shelf, the area of the displayed products is calculated from the image of the product shelf by object detection processing, and when the ratio of the area of the displayed products to the area of one shelf section area is equal to or lower than the threshold value, it is determined to be out of stock or out of stock.

[0091] As described above, in the product display state detection system according to this Modification 2, each product shelf at the opening of the store is imaged and compared with the image captured by the camera of the cart during business hours, so that changes in the display state such as out-of-stock, out-of-stock or mess of products on each product shelf can be detected. Therefore, even when the store is crowded, it is possible to smoothly and efficiently detect changes in the display state of the products on the product shelf.

[0092] Note that each configuration illustrated in the above embodiments and modifications is functionally schematic, and it is not necessarily physically configured as illustrated. That is, the form of distribution and integration of each device is not limited to that illustrated, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage situations, etc.

Industrial Applicability

[0093] The product display state detection system, management device, product display state detection method, and product display state detection program according to the present invention are suitable for smoothly and efficiently detecting the display state of products on a product shelf even when the store is crowded.

Explanation of Signs

[0094] 10 Cart 20 Camera 30 Store clerk terminal 40 Management device 41 Display unit 42 Input unit 44 Communication unit 45 Storage unit 45a Product management data 45b Stock threshold data 45c Inventory Data 45d Merchandise Shelf Data 45e Detection Data 46 Control Unit 46a Merchandise Management Section 46b Inventory Threshold Setting Section 46c Inventory Management Section 46d Merchandise Shelf Management Section 46e Detection Section 46f Status Judgment Section 46g Merchandise Identification Section 46h Inventory Judgment Section 46i Notification Section 50 Wireless Router 90 Shopping Basket 100 Management Device 110 Memory Unit 111 Initial Display State Data 120 Control Unit 121 Initial Display State Acquisition Section 122 Status Judgment Section

Claims

1. An imaging unit installed on a cart for a customer to store goods, A detection unit that detects the display state of goods on a product shelf in a store from an image of the product shelf captured by the imaging unit, An informing unit that informs predetermined product information according to the display state of the goods on the product shelf detected by the detection unit Comprising The imaging unit is installed so as to be able to image the lateral position of the cart with respect to the traveling direction of the cart Product display state detection system.

2. The informing unit The product display state detection system according to claim 1, wherein when the display state of the goods on the product shelf is an out-of-stock state or a shortage state, the product information is informed.

3. The informing unit The product display state detection system according to claim 1, wherein when the display state of the goods on the product shelf is disordered, the product information is informed.

4. Further comprising a first storage unit that stores inventory information of goods in a store, The informing unit When the display state of the goods on the product shelf is an out-of-stock state or a shortage state, referring to the inventory information of the goods stored in the first storage unit, and when the inventory number of the out-of-stock state or shortage state goods is equal to or more than a predetermined threshold value, notifying a replenishment request for the goods, and when the inventory number of the out-of-stock state or shortage state goods is less than a predetermined threshold value, not notifying a replenishment request for the goods The product display state detection system according to any one of claims 1 to 3.

5. Further comprising a second storage unit that stores shelf division information associating the identification information of the product shelf with the type of goods displayed on the product shelf, The detection unit Detecting the identification information of the product shelf from the image of the product shelf, and when the goods on the product shelf are in an out-of-stock state or a shortage state, referring to the shelf division information stored in the second storage unit to specify the type of goods associated with the identification information of the product shelf The product display state detection system according to any one of claims 1 to 3.

6. Further comprising a third storage unit that stores a reference image of a product shelf captured in advance, The detection unit Based on the image of the product shelf captured by the imaging unit and the reference image of the product shelf stored in the third storage unit, detecting a change in the display state as the display state of the goods on the product shelf The product display state detection system according to any one of claims 1 to 3.

7. A management device in a product display state detection system for detecting the display state of goods on a product shelf in a store, A detection unit that is installed on a cart for a customer to store goods and detects the display state of goods on a goods shelf from an image of the goods shelf captured by an imaging unit capable of imaging a side position of the cart with respect to the traveling direction of the cart; A notification unit that notifies predetermined product information according to the display state of the products on the product shelf detected by the detection unit A management device comprising the above.

8. A method for detecting the display state of products on a store's product shelf, which is a method for detecting the display state of products in a product display state detection system for detecting the display state of products on a store's product shelf, A detection step of detecting the display state of products on the product shelf from an image of the product shelf captured by an imaging unit that is installed on a cart for a customer to store products and is capable of imaging a side position of the cart with respect to the traveling direction of the cart; A notification step of notifying predetermined product information according to the display state of the products on the product shelf detected in the detection step Including A method for detecting the display state of products.

9. A product display state detection program executed by a management device in a product display state detection system for detecting the display state of products on a store's product shelf, A detection procedure for detecting the display state of products on the product shelf from an image of the product shelf captured by an imaging unit that is installed on a cart for a customer to store products and is capable of imaging a side position of the cart with respect to the traveling direction of the cart; A notification procedure for notifying predetermined product information according to the display state of the products on the product shelf detected by the detection procedure A product display state detection program that executes the above.

Images