Management device, management method, and program

Abstract

A management device (100) is provided with: a surface area calculating unit (102) for acquiring the surface area of a displayable region of a display shelf on which products can be displayed, and calculating, for the display shelf, the surface area ratio between the surface area of a non-displayed region in which products are not displayed, detected from a captured image, and the surface area of the display shelf; a determining unit (104) for determining whether to issue a notification, in accordance with the surface area ratio; and a notifying unit (106) for issuing a notification if it has been determined that a notification is to be issued.

Description

[Technical field]

[0001] The present invention relates to a management device, a management method, and a program, and more particularly to a management device, a management method, and a program for managing display shelves in a store. [Background technology]

[0002] Patent Document 1 describes capturing an image of the product shelf status and superimposing an image (Fig. 4) that is color-coded according to the display status so that the display shortage can be identified. Patent Document 2 describes sending a notification to replenish products when there are few products on the product shelf, and making it possible to place a reorder to keep the stock in stock. Other examples include a system that uses a robot to track the layout of product shelves (Patent Document 3) and an inventory monitoring system that monitors product inventory and notifies the user to replenish (Patent Document 4). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-58105 A [Patent Document 2] Special Publication No. 2010-517148 [Patent Document 3] Special Publication No. 2019-523925 [Patent Document 4] JP 2019-152934 A Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, the problem of securing employees in stores due to labor shortages is becoming more serious. Under these circumstances, there is a need to reduce the labor required for inventory management and replenishing shelves, thereby reducing the burden on employees.

[0005] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique for improving the efficiency of the task of replenishing products on display shelves. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, each aspect of the present invention employs the following configuration.

[0007] The first aspect relates to a management device. A management device according to a first aspect of the present invention includes: an area calculation means for acquiring an area of ​​a displayable area in which products can be displayed on a display shelf, and calculating an area ratio of an area of ​​a non-display area in which the products are not displayed, detected from a captured image of the display shelf, to the area of ​​the display shelf; a determination means for determining whether or not to issue a notification in accordance with the area ratio; and a notification means for, when it is determined that the notification should be made, making the notification.

[0008] A second aspect relates to a management method implemented by at least one computer. The management method according to the second aspect comprises: The management device: Obtaining the area of ​​a displayable area in which the products on the display shelf can be displayed; calculating an area ratio of a non-display area in which the product is not displayed, detected from the captured image of the display shelf, to the area of ​​the display shelf; determining whether or not to issue a notification according to the area ratio; When it is determined that the notification should be performed, the notification is performed.

[0009] Another aspect of the present invention may be a program for causing at least one computer to execute the method of the second aspect, or a computer-readable recording medium having such a program recorded thereon. This recording medium includes a non-transitory tangible medium. The computer program comprises computer program code which, when executed by a computer, causes the computer to perform the management method on the management device.

[0010] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention.

[0011] Furthermore, the various components of the present invention do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be a part of another component, or part of one component may overlap with part of another component, etc.

[0012] In addition, although the method and computer program of the present invention describe a number of steps in a sequential order, the order does not limit the order in which the steps are executed. Therefore, when implementing the method and computer program of the present invention, the order of the steps can be changed as long as it does not affect the content.

[0013] Furthermore, the steps of the method and computer program of the present invention are not limited to being executed at different timings, and therefore a step may occur while another step is being executed, or the execution timing of a step may overlap partially or entirely with the execution timing of another step, etc. Effect of the Invention

[0014] According to each of the above aspects, it is possible to provide a technique for improving the efficiency of the task of replenishing products on display shelves. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram conceptually illustrating an example of the configuration of a store system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining an image captured by a camera. [Diagram 3] FIG. 4 is a diagram showing an example of a data structure of image information for each image to be stored. [Figure 4] 2 is a functional block diagram logically illustrating a configuration of a management device according to an embodiment of the present invention; FIG. [Diagram 5] FIG. 13 is a diagram illustrating an example of a data structure of shelf area information. [Figure 6] 11 is a diagram illustrating an example of a data structure of notification criteria information indicating a determination criterion. [Figure 7] FIG. 13 is a diagram showing an example of a data structure of display status information. [Figure 8] 5 is a block diagram illustrating a hardware configuration of a computer that realizes the management device illustrated in FIG. 4. [Figure 9] 5 is a flowchart illustrating an example of an operation of the management device of the present embodiment. [Figure 10] FIG. 13 is a diagram showing an example of an image of a display shelf at a given time. [Figure 11] FIG. 11 is a diagram showing an example of a notification screen by a notification unit of the present embodiment. [Figure 12] FIG. 2 is a functional block diagram illustrating an example of a logical configuration of a management device according to the present embodiment. [Figure 13] FIG. 13 is a diagram for explaining a method for setting a judgment criterion. [Figure 14] 13 is a diagram showing an example of an area shape of a determination criterion in a shelf image area. FIG. [Figure 15] 11 is a diagram illustrating an example of a data structure of notification criteria setting information. [Figure 16] FIG. 2 is a functional block diagram illustrating an example of a logical configuration of a management device according to the present embodiment. [Figure 17] 10A and 10B are diagrams illustrating an example of a data structure of product display information and product information. [Figure 18] FIG. 11 is a functional block diagram showing another example of a partial logical configuration of the management device of the present embodiment. [Figure 19] FIG. 11 is a functional block diagram showing another example of a partial logical configuration of the management device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals, and the description will be omitted as appropriate. In the following drawings, the configuration of the parts that are not related to the essence of the present invention is omitted and is not shown.

[0017] In the embodiment, "acquisition" includes at least one of the following: a device goes to retrieve data or information stored in another device or storage medium (active acquisition), and inputs data or information output from another device to the device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0018] (First embodiment) <System Overview> 1 is a block diagram conceptually illustrating an example of the configuration of a store system 1 according to an embodiment of the present invention. The store system 1 includes a management device 100, a camera 5, a user terminal 20, and an image processing device 30. The management device 100, the camera 5, the user terminal 20, and the image processing device 30 are connected via a communication network 3. The image processing device 30 may be provided in the same hardware as the management device 100, or may be provided outside the management device 100 as separate hardware.

[0019] A store is provided with a display shelf 10 for displaying merchandise, including showcases. The store is also provided with a surveillance camera 5 for capturing images of the inside of the store. For example, the camera 5 may be a camera capable of capturing images in all directions of 360 degrees, installed on the ceiling of the store for crime prevention purposes. As an example, the camera 5 is a camera equipped with a fisheye lens.

[0020] The camera 5 includes an imaging element such as a lens and a CCD (Charge Coupled Device) image sensor. For example, the camera 5 is a network camera such as an IP (Internet Protocol) camera. The network camera has, for example, a wireless LAN (Local Area Network) communication function, and is connected to the image processing device 30 via a communication network, that is, a relay device (not shown) such as a router. The camera 5 also includes a camera body and lens The camera may be equipped with mechanisms for controlling the orientation, zoom control, focusing, etc.

[0021] The user terminal 20 is a smartphone, a tablet terminal, a personal computer, etc. In this embodiment, the user terminal 20 is a terminal used by a user U such as a store clerk, a manager of a store or a management center, etc.

[0022] The management device 100 is a device that manages the status of display shelves 10 for products in a store. A display shelf 10 is a fixture that displays products in a store, and is configured by providing at least one shelf board (hereinafter also referred to as a shelf 14) on a basic outer frame 12, and is also called a gondola. Here, one display shelf 10 is configured by providing multiple shelves 14 on one basic outer frame 12.

[0023] The state of the display shelves 10 refers to the display state of the products 16 for each display shelf 10 or each shelf 14, and in particular refers to a shortage state of the products 16. The management device 100 acquires an image of the display shelves 10 from the video captured by the camera 5, and detects and manages the state of the display shelves 10, for example, a shortage of the products 16.

[0024] Fig. 2 is a diagram for explaining images captured by camera 5. Fig. 2(a) is a diagram showing an example of an image captured by camera 5 from above the entire store. Fig. 2(b) is a diagram showing an example of an image captured by camera 5 of display shelf 10 from the front direction. Fig. 2(c) is a diagram showing an example of an image of one display shelf 10 extracted from the image of Fig. 2(b).

[0025] An image 200 (FIG. 2(a)) captured by the camera 5 is transmitted to the image processing device 30. The image 200 transmitted to the image processing device 30 is a video captured in real time by the surveillance camera 5, or a still image captured periodically or at a specified time. However, the image 200 transmitted to the image processing device 30 does not have to be transmitted directly from the camera 5, and may be an image recorded on a recording medium (not shown). Furthermore, the image 200 transmitted to the image processing device 30 may be an image delayed by a specified time.

[0026] The image 210 in FIG. 2(b) is an image captured in a certain imaging direction, and the image 210 includes three display shelves 10. t An image 212 of each display shelf 10 is cut out from among the display shelves 10, and the curved image is corrected to convert it into an image 230 (FIG. 2(c)) of the display shelf 10 viewed from the front. The management device 100 uses this image-processed image 230 of the display shelf 10 to monitor the state of the display shelf 10.

[0027] The image 230 of the display shelf 10 processed by the image processing device 30 is stored in the storage device 110. FIG. 3 is a diagram showing an example of the data structure of image information 240 of each image 230 stored. The image information 240 includes an image ID for identifying the image 230 stored in the storage device 110, the image capture date and time of the image 230, and the position and identification information (display shelf ID) of the display shelf 10 identified from the captured image. The image ID may be a path indicating the file name and storage location of the image 230. The image capture date and time may be the frame number of the image 230. The display shelf 10 in the captured image may be identified by the image processing device 30 from information such as the shooting direction by the camera 5. Alternatively, the image processing device 30 may identify the display shelf 10 by image recognition of a marker for identifying the display shelf 10 that is provided in advance on the display shelf 10. There are various possible methods for identifying the display shelf 10 from the captured image, and the method is not limited to this method.

[0028] <Example of functional configuration> 4 is a functional block diagram logically showing a configuration of a management device 100 according to an embodiment of the present invention. The management device 100 includes an area calculation unit 102, a determination unit 104, and a notification unit 106. The management device 100 further includes a storage device 110. The storage device 110 may be provided inside the management device 100 or may be provided externally. The storage device 110 may physically be a plurality of storage devices.

[0029] In the storage device 110, for example, the area of ​​a displayable area in which products can be displayed on a display shelf is stored in advance.

[0030] The area calculation unit 102 acquires the area of ​​the displayable area, and calculates the area ratio of the area of ​​the non-display area of ​​the display shelf 10, where the products 16 detected from the captured image are not displayed, to the area of ​​the display shelf 10. The determination unit 104 determines whether or not to issue a notification according to the area ratio calculated by the area calculation unit 102. The notification unit 106 issues a notification when it is determined that a notification should be issued.

[0031] 5 is a diagram showing an example of the data structure of the shelf area information 302. The shelf area information 302 is stored in advance on each shelf 14 of each display shelf 10 in the store. Area The value of the area is stored as setting information in the storage device 110. This setting information can be referenced by the operator and can be used by the store. of This can be changed at any time in accordance with changes in the layout of the display shelves 10 and shelves 14.

[0032] Here, the "area of ​​the displayable area" is indicated by the number of squares of the plane of the displayable area of ​​shelf 14, but is not limited to this and may be each dimension indicating the shape of the displayable area on shelf 14. For example, if the shape of the displayable area of ​​shelf 14 is rectangular, it may be the width and depth dimensions of the displayable area of ​​shelf 14. In other words, any information that can determine the area of ​​the displayable area of ​​shelf 14 will suffice.

[0033] In the example of FIG. 5(a), the shelf area information 302 stores a shelf ID and the area of ​​the product displayable area of ​​the shelf 14 in association with each other. The shelf ID in this example includes identification information for identifying each display shelf 10 installed in the managed store and each shelf 14 of the display shelf 10. In the example of FIG. 5(b), the shelf area information 302 stores a store ID, a display shelf ID, a shelf ID, and the area of ​​the product displayable area in association with each other. Remember The display shelf ID may include, for example, information indicating the installation position of the display shelf 10 in the store. The position of the display shelf 10 in the image captured by the camera 5 in the store can be identified from the image capture position in the store, and the corresponding display shelf 10 can be identified. Furthermore, in the example of FIG. 5(c), the shelf area information 302 may be stored by associating the area of ​​the product displayable area of ​​the shelf 14 with the position information of the display shelf 10 or each shelf 14 of the display shelf 10. The position information may be relative position information indicating the relative position of each display shelf 10 in the store, or absolute position information indicated by longitude and latitude, and may be any information capable of identifying the shelf 14. The position of each shelf 14 may be information indicating which level of the display shelf 10 the shelf 14 is located, or information indicating the height at which the shelf 14 is installed.

[0034] The area calculation unit 102 calculates an area ratio of the area of ​​the detected non-display area to the area of ​​the displayable area of ​​each shelf 14. Specific examples of the calculation method of the area ratio will be described below, but the method is not limited to these. Specifically, for example, the area calculation unit 102 first detects a shelf area from the image 230 by image recognition processing using the image processing device 30, and specifies the area of ​​a portion of the shelf area on which no product 16 is placed. Then, the area calculation unit 102 obtains the area of ​​the shelf area of ​​the shelf 14 from the shelf area information 302 in FIG. 5, and calculates the area ratio.

[0035] In addition, the area of ​​each region does not necessarily have to be specified, and the ratio within the image is calculated. hand For example, the area calculation unit 102 detects the shelf area and specifies, among the shelf areas of each shelf 14, an area on which no products 16 are placed as a non-display area. Then, the area calculation unit 102 obtains the ratio of the non-display area to the shelf area in the shelf image, and sets this as the area ratio.

[0036] Also, as an example, the determination unit 104 determines to perform a notification when the non-display area is equal to or greater than a threshold (for example, 80%), and determines not to perform a notification when the non-display area is less than the threshold.

[0037] FIG. 6 is a diagram showing an example of a data structure of notification criteria information 310 indicating the judgment criteria. In this example, a plurality of notification levels are associated with an area ratio serving as the judgment criteria. A plurality of notification levels can be set, and the notification method and notification content can be set for each level. In this example, level 1 and level 2 are set. If the non-display area is 40% or more, the determination unit 104 determines the notification level to be 1. If the non-display area is 20% to 40%, the determination unit 104 determines the notification level to be 2. However, the judgment criteria value is an example and is not limited to this.

[0038] At notification level 2, for example, a notification may be sent to the user terminal 20 of a store clerk instructing the clerk to replenish the product 16. At notification level 1, a notification may also be sent to the user terminal 20 of a manager of the store and management center, informing the user that the product 16 is out of stock and instructing the clerk to urgently replenish the product 16. Possible notification methods include, but are not limited to, sending a message to the user terminal 20, displaying a pop-up screen, vibrating, outputting a voice, and the like.

[0039] The determination result by the determination unit 104 is stored in the storage device 110 as display state information 320. Fig. 7 is a diagram showing an example of the data structure of the display state information 320. The display state information 320 stores, for each shelf, a shelf ID, a date and time (date and time when the image was captured, date and time of determination, etc.), an area ratio, a notification flag (for example, "1" if notification is to be performed, "0" if notification is not to be performed, etc.), and an image ID of the captured image of the shelf 14 (for example, corresponding to the image ID of the image information 240) in association with each other.

[0040] <Hardware configuration example> Fig. 8 is a block diagram illustrating a hardware configuration of a computer 1000 that realizes the management device 100 shown in Fig. 4. The computer 1000 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0041] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to a bus connection.

[0042] The processor 1020 is implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0043] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0044] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the computer 1000. The processor 1020 loads each of these program modules onto the memory 1030 and executes them to realize each function corresponding to the program module. The storage device 1040 also stores each piece of information stored in the storage device 110.

[0045] The program module may be recorded in a recording medium. The recording medium for recording the program module may include a non-transitory tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0046] The input / output interface 1050 is an interface for connecting the computer 1000 to various input / output devices.

[0047] The network interface 1060 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1060 to the communication network may be a wireless connection or a wired connection.

[0048] The computer 1000 is connected to necessary devices (for example, a display device such as a touch panel, an input device, a camera 5, a microphone, and a speaker) via an input / output interface 1050 or a network interface 1060.

[0049] Similarly, the user terminal 20 that receives a notification from the management device 100 can also be realized by a computer 1000 having the configuration shown in FIG.

[0050] In this embodiment, an application for receiving notifications from the management device 100 is installed in advance on the user terminal 20. Alternatively, an application that can connect to the Internet, such as a browser, and access a user page of a website provided by the management device 100 may be installed in the user terminal 20. The user page can be accessed by logging in by inputting authentication information previously registered by the user. Information indicating the content of the notification may then be presented on the user page.

[0051] figure 4 Each component of the management device 100 of this embodiment is realized by any combination of hardware and software of the computer 1000 in Fig. 8. Those skilled in the art will understand that there are various variations in the realization method and device. The functional block diagram showing the management device 100 of each embodiment described below shows logical functional blocks rather than a hardware-based configuration.

[0052] <Example of operation> FIG. 9 is a flowchart showing an example of the operation of the management device 100 of this embodiment. First, in the management device 100, the area calculation unit 102 detects, using the image processing device 30, a non-display area in which no product 16 is displayed on the display shelf 10 from the captured image 230 (step S101).

[0053] Fig. 10 is a diagram showing an example of an image 230 of the display shelf 10 at a specific time. For example, Fig. 10(a) is an image 230 of the display shelf 10 taken at 8:30, and Fig. 10(b) is an image 230 of the display shelf 10 taken one hour later at 9:30. At 8:30, the display shelf 10 is completely filled.

[0054] Next, the area calculation unit 102 obtains the area of ​​the displayable area of ​​the shelves 14 of the display shelf 10, and calculates the area ratio of the non-display area of ​​the shelves 14 to the displayable area (step S103). At 8:30 in FIG. 10(a), the area ratio of the non-display area to the displayable area of ​​each shelf 14 is 0%. Therefore, the determination unit 104 determines that no notification should be made since the area ratio does not satisfy the notification criterion (NO in step S105), and bypasses step S107 to end this process.

[0055] In step S103, the area calculation unit 102 acquires the area of ​​the displayable region of the shelf 14 of the display shelf 10 stored in the storage device 110, but is not limited to this. For example, an input from the user U may be accepted, or the area may be calculated and acquired from the captured image 230 based on information such as markers provided on the shelf 14.

[0056] On the other hand, at 9:30 in FIG. 10(b), in step S101, the area calculation unit 102 detects the non-display area of ​​each shelf 14 using the image processing device 30, and calculates the area ratio of each shelf 14 (step S103). As a result, the area ratio of the non-display area to the displayable area of ​​the shelf 14 is, for example, between 20% and 40% for shelf ST2, the area ratio of shelf ST3 is 40% or more, and the area ratio of the other shelves is less than 20% (step S103). As a result of the determination by the determination unit 104 as to whether the area ratio of each shelf 14 satisfies the notification criterion, shelf ST2 is determined to be at notification level 2 (YES in step S105), shelf ST3 is determined to be at notification level 1 (YES in step S105), and the other shelves are determined to be not notified (NO in step S105).

[0057] Therefore, for shelves ST2 and ST3, the process proceeds to step S107, and the notification unit 106 issues a notification corresponding to each notification level. For the other shelves, step S107 is bypassed and this process ends.

[0058] Fig. 11 is a diagram showing an example of a notification screen by the notification unit 106 of this embodiment. A notification screen 400 in Fig. 11(a) includes a shelf list tab 402 and a notification list tab 410. The shelf list tab 402 displays, for example, a list (not shown) of all display shelves 10 in each store. The notification list tab 410 displays a list of notification contents by the notification unit 106. The list includes, for each notification, a thumbnail image 412, a notification content display section 414, and a notification date and time display section 416.

[0059] The thumbnail image 412 is a thumbnail image of the front image 230 of the shelf 14 that is the subject of the notification. The notification content display section 414 may display the identification information of the shelf 14 and the notification level. Alternatively, a message such as "Shelf AAA: 35% out of stock" may be displayed, indicating that the shelf 14 with the identification information "Shelf AAA" (which may be a name for displaying the notification (e.g. "Lunch Box")) is out of stock, and Area calculation unit 102 may display the value of the area ratio of the non-display area calculated by the above. The notification content display unit 414 is merely an example and is not limited to this. For example, the notification content display unit 414 may display the notification content using icons for each notification level or a bar graph showing the area ratio. The notification date and time display unit 416 displays the date and time when the image 230 of the shelf 14 determined to require notification was captured or the date and time of determination.

[0060] Furthermore, by selecting each notification field in FIG. 11(a), the screen transitions to a detailed contents screen 420 (FIG. 11(b)) for each notification. In the example of FIG. 11(b), the detailed contents screen 420 includes a notification date and time display section 422, a shelf image display section 424, a confirmation button 426, and a back button 428. The notification date and time display section 422 displays the date and time when the image 230 of the shelf 14 for which it has been determined that a notification is necessary was captured, or the determination date and time. The shelf image display section 424 displays the image 230 of that shelf 14, and in that image 230, the shelf 14 is shown in which the product 16 is out of stock. The confirmation button 426 is used to accept an operation indicating that the user U of the user terminal 20 has confirmed the notification. R A back button 428 accepts an operation to return to the display of the notification list tab 410 screen in FIG.

[0061] As described above, in this embodiment, the area calculation unit 102 calculates the area ratio of the area of ​​the non-display area to the area of ​​the display area of ​​each shelf 14, and the notification unit 106 issues a notification for the shelf that is determined by the determination unit 104 to issue a notification based on the area ratio.

[0062] In this way, according to this embodiment, it is possible to detect a shortage of product 16 on each shelf 14 of each display shelf 10 in the store from the image 230 of the camera 5 and to notify the shortage.

[0063] Second embodiment 12 is a functional block diagram showing an example of a logical configuration of the management device 100 of this embodiment. The management device 100 of this embodiment is similar to the above embodiment except that the management device 100 performs notification determination according to the position of the non-display area in addition to the area ratio of the non-display area. The management device 100 of this embodiment further includes a position identification unit 120 in addition to the configuration of the management device 100 of FIG.

[0064] The position specifying section 120 specifies the position of the non-display area from the captured image 230. The determining section 104 determines to issue a notification when the area ratio meets the criterion for issuing a notification and the position of the non-display area meets the criterion.

[0065] The location of the area may be indicated, for example, by the location or sales area of ​​the display shelf 10 within the store, the type of product 16 displayed on the display shelf 10, the position of the shelf 14 within the display shelf 10 (number of levels, center above or below, etc.), or the layout within the shelf 14.

[0066] Fig. 13 is a diagram for explaining a method of setting judgment criteria when the position of an area is set for each shelf 14. In the example of Fig. 13(a), a shelf image area 500 of each shelf 14 is cut out from a captured image 230 of the display shelf 10 and divided into nine areas R1 to R9.

[0067] In the first example, as shown in FIG. 13(b), the notification area reference information 330 specifies an area (for example, at least area R1 and area R4) for which a notification is to be sent when the area is determined to be a non-display area. In other words, when there are no products 16 from the front to the back of the shelf 14 (at least area R1 and area R4), the product 16 that should be displayed in that area is out of stock. stomach It is determined that.

[0068] Here, whether or not each area is included in the non-display area is determined by determining that the area is included in the non-display area if the proportion of the area in which product 16 is not detected exceeds a predetermined proportion of the area (e.g., 90%).

[0069] In a second example, the notification area reference information 330 may specify an area (for example, at least one of the area R7, area R8, and area R9 in the back row of the shelf 14) for which notification is not performed (notification OFF) if the area is determined to be a non-display area. of If there are no products 16 in the back row of shelf 14 (for example, at least one of area R7, area R8, and area R9), then product 16 is placed at the front of shelf 14, and no notification is required.

[0070] Furthermore, the areas may be prioritized, and the determination and notification may be performed with priority given to areas with higher priorities. For example, when there is no product 16 in an area where the product 16 should be displayed, such as the front row or center of the shelf 14, it is preferable to perform a notification to move the product 16 to the front. For this reason, as shown in Fig. 13(c), the notification priority information 332 specifies the areas to be notified with priority. The determination unit 104 determines a combination of areas with higher priorities with priority according to the priorities in the notification priority information 332.

[0071] The method of giving priority includes, for example, shortening the cycle of the determination process or raising the notification level as the priority level increases.

[0072] Also, the position of the non-display area may be stored in the form of the area shape, and the judgment criterion may be whether or not the degree of agreement with the shape of the non-display area is equal to or greater than a predetermined percentage.

[0073] FIG. 14 is a diagram showing an example of the area shape of the judgment criteria in the shelf image area 500. FIG. 14(a) shows the shape 512 of the area with the first priority in the notification priority information 332 of FIG. 13(c). Furthermore, the positions of the area R2 and the area R5 included in the shape 512 in the shelf image area 500 are shown. FIG. 14(b) shows the shape 522 or the shape 524 of the area with the second priority in the notification priority information 332 of FIG. 13(c). Furthermore, the positions of the areas R1 and R4, or the areas R3 and R6 included in the shape 522 or the shape 524 in the shelf image area 500 are shown. FIG. 14(c) shows the shape 532, the shape 534, and the shape 536 of the area with the third priority in the notification priority information 332 of FIG. 13(c). The positions of the areas R1, R2, and / or R3 included in each shape in the shelf image area 500 are shown.

[0074] Each shape that serves as a judgment criterion is stored in the storage device 110 for each shelf 14. The position identification unit 120 identifies the shape of the non-display area detected from the image 230. The judgment unit 104 then uses the image processing device 30 to compare the shape of the identified non-display area with the shapes stored in the storage device 110 and calculates the degree of match. If the degree of match is equal to or greater than a predetermined rate (e.g., 80%), the judgment unit 104 determines that a notification should be issued. If the degree of match is less than the predetermined rate, the judgment unit 104 determines that a notification should not be issued.

[0075] In addition, the determination unit 104 determines whether or not a predetermined position The threshold value is changed for the display shelf 10, and the changed threshold value is used to determine whether or not to issue a notification. These threshold values ​​are stored in the storage device 110 as notification criterion setting information 340.

[0076] FIG. 15 is a diagram showing an example of the data structure of the notification criteria setting information 340. The notification criteria setting information 340 may be set for each display shelf 10, each shelf 14, or each type of product 16 to be displayed. For example, the threshold value of the shelf 14 of the display shelf 10 where lunchboxes are displayed is changed to a lower setting value (e.g., 40 to 20%) than the standard threshold value during the lunch time slot compared to the display shelves 10 and shelves 14 where other products 16 are displayed. This is to perform an early notification so as to prevent a shortage during the lunch time slot. In addition, the threshold value is changed to a higher setting value (e.g., 90% or more) than the standard threshold value for the shelf 14 of lunchboxes immediately after lunch or during the nighttime. This is because there are fewer customers at night, so a shortage of lunchboxes is not a problem.

[0077] Furthermore, the threshold value may be changed according to season, month, week, day of the week, specific period, specific date and time, etc., other than the time period. For example, different threshold values ​​may be set for weekdays and weekends in the notification criteria setting information 340. Furthermore, the notification criteria setting information 340 may include a threshold value that is different from the standard for a specific product 16 during a campaign period. Threshold may be set.

[0078] Furthermore, in the example of FIG. 15, only the criteria for whether or not to notify are specified, but as described above, the thresholds for each notification level may be changed depending on the time period, etc.

[0079] Furthermore, the determination unit 104 may determine, based on the position of the non-display area, that a notification should be issued unconditionally if no product 16 is displayed in a specific position on the display shelf 10.

[0080] In this example, for example, if the product 16 is not displayed in the position with the highest priority (area R2 and area R5) described in FIG. 14, the judgment unit 104 can determine to issue a notification, bypassing judgment processes based on other judgment criteria. By designating a position where monitoring of out-of-stock status is particularly important, the notification can be given top priority. The specific position can be designated, for example, as a specific display shelf 10 in a store, a specific shelf 14 within the display shelf 10, a specific area within the shelf 14, or either the display shelf 10 or the shelf 14 on which the specific product 16 is displayed. The designated specific position information is stored in the storage device 110. Furthermore, the time zone, date and time, period during which the designation is valid, and the like may be designated in association with the time zone.

[0081] As described above, in this embodiment, the criteria for determining whether or not to issue a notification can be set and changed in various ways, so that notifications can be issued efficiently based on criteria that meet the needs of the store or other on-site location and the company policy of the administrator.

[0082] Third embodiment 16 is a functional block diagram showing an example of a logical configuration of the management device 100 of this embodiment. The management device 100 of this embodiment is similar to at least one of the other embodiments described above, except that it has a configuration for identifying products 16 to be displayed in the detected non-display area and presenting information about the products. The management device 100 of this embodiment can be combined with at least one of the other embodiments described above to the extent that there is no contradiction.

[0083] 12, the management device 100 further includes a product identification unit 122 and a display processing unit 130. The product identification unit 122 identifies a product 16 that was displayed in a non-display area of ​​the display shelf 10 based on the position of the non-display area on the display shelf 10 identified by the position identification unit 120. The display processing unit 130 causes the display device 132 to display text information and an image related to the identified product 16.

[0084] The display device 132 may be a display device of the user terminal 20, or may be a terminal device (not shown) of a management center. Also, the display device 132 may be presented on a web page that can be viewed by logging in to a website provided by the management device 100 from the user terminal 20.

[0085] For example, the text information regarding the product 16 is information indicating the name, product code, price, etc. of the product 16. Furthermore, the text information may include a uniform resource locator (URL) of a web page introducing the product 16.

[0086] 17 is a diagram showing an example of the data structure of the product display information 350 and the product information 352. The product display information 350 associates a store ID, a display shelf ID, a shelf ID, position information, and a product ID. In other words, the product display information 350 stores position information on where the products 16 are displayed for each store, each display shelf 10, and each shelf 14.

[0087] The product information 352 is associated with a product ID, a product name, image information, and product details. The product ID is not particularly limited as long as it is identification information that can identify the product, and is, for example, a product code. The image information includes at least the image file name of the image of the product and the path to the location where the image file is saved. The product details may include information such as the price, the distributor, and the URL (Uniform Resource Locator) of the website for product introduction information.

[0088] Information on each product in the product information 352 may be linked to information on the display location of the product (product display information 350).

[0089] The notification unit 106 may then notify the user that the product 16 is out of stock, together with information about the display position of the product 16 identified by the product identification unit 122. When the user terminal 20 receives the notification, the user U can use the user terminal 20 to view the information about the out-of-stock product 16 displayed by the display processing unit 130, and efficiently replenish the shelf 14 of the notified display shelf 10 with the product 16.

[0090] Furthermore, as shown in FIG. 18, the management device 100 may further include a quantity calculation unit 124 in addition to the configuration of FIG. 16. The quantity calculation unit 124 calculates the quantity of the identified products 16 based on the area of ​​the non-display area. The product information 352 in FIG. 17(b) further includes information on the size of the products (for example, the area occupied by one product when displayed). First, the quantity calculation unit 124 calculates the area in which the products 16 were displayed within the detected non-display area. Then, the quantity calculation unit 124 calculates the quantity of the products 16 that are out of stock from the area of ​​the calculated area and the area occupied by the products.

[0091] This also clarifies the quantity of products 16 to be replenished, making it possible to carry out the replenishment work efficiently.

[0092] According to this embodiment, the same effects as those of the above embodiment can be achieved, and furthermore, since it is possible to identify the products 16 that were displayed in the detected non-display area, it is possible to notify the user U of the products 16 that are out of stock on the shelves 14 of the display shelf 10, and the user U can efficiently replenish the out-of-stock products 16.

[0093] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted. The settings such as the thresholds and areas that serve as the judgment criteria described in the above embodiment may be specified by store clerks or managers for each store, but they may also be managed by a management center, which can specify the setting values ​​for each store.

[0094] 19, the management device 100 may further include an order processing unit 140 in addition to the configuration of FIG. 18. The order processing unit 140 can perform an order process for the product 16 that is out of stock, based on the product 16 identified by the product identification unit 122 and the quantity of the product 16 calculated by the quantity calculation unit 124.

[0095] In this configuration, information such as the stock amount of each product 16 in the back yard, the stock amount to be ordered, and the order amount is stored in advance in the storage device 110 as stock information 354. Then, the order processing unit 140 refers to the stock information 354 from the calculated quantity of the product 16 to determine whether or not it is necessary to order the product 16, and calculates the order amount if it is determined that an order is necessary. The order processing unit 140 may perform the order processing automatically. Alternatively, the order processing unit 140 may notify the user terminal 20 of the administrator of the order contents, and execute the order processing by receiving an instruction to execute the order processing or an operation to change the order contents from the administrator.

[0096] According to this configuration, not only can the out-of-stock state be notified, but also an order process for the out-of-stock item can be carried out based on the inventory status.

[0097] In addition, the setting and changing of judgment criteria can also be optimized using machine learning.

[0098] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above-mentioned embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In the present invention, when information about a user is acquired and used, this shall be done lawfully.

[0099] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An area calculation means for acquiring an area of ​​a displayable area in which products can be displayed on a display shelf, and calculating an area ratio of an area of ​​a non-display area in which the products are not displayed, detected from a captured image of the display shelf, to the area of ​​the display shelf; a determination means for determining whether or not to issue a notification in accordance with the area ratio; a notification means for, when it is determined that the notification should be performed, performing the notification. 2. In the management device according to 1., The display device further includes a position specifying means for specifying a position of the non-display area from the captured image, The determination means determines that the notification should be made when the area ratio meets the criterion for making the notification and the position of the non-display area meets the criterion. 3. In the management device according to 1. or 2., The determination means determines that the notification should be performed when the area ratio is equal to or greater than a threshold value. 4. In the management device according to 3., The determination means determines whether or not to change the threshold value for the display shelf at a predetermined position and to issue the notification depending on a time period. 5. In the management device according to any one of 1. to 4., The display device further includes a position specifying means for specifying a position of the non-display area from the captured image, The determination means determines, based on the position of the non-display area, that the notification should be made unconditionally if the product is not displayed in a specific position on the display shelf. 6. In the management device according to any one of 1. to 5., A position specifying means for specifying a position of the non-display area from the captured image; a product identification means for identifying a product that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The management device further comprises a display processing means for displaying text information and an image relating to the identified product. 7. In the management device according to any one of 1. to 6., A position specifying means for specifying a position of the non-display area from the captured image; a product identification means for identifying a product that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The management device further comprises a quantity calculation means for calculating a quantity of the identified products based on the area of ​​the non-display area.

[0100] 8. The management device: Obtaining the area of ​​a displayable area in which the products on the display shelf can be displayed; calculating an area ratio of a non-display area in which the product is not displayed, detected from the captured image of the display shelf, to the area of ​​the display shelf; determining whether or not to issue a notification according to the area ratio; When it is determined that the notification should be given, the notification is given. 9. In the management method described in 8., The management device further comprises: Identifying a position of the non-display area from the captured image; A management method that determines that the notification should be made when the area ratio meets a criterion for making the notification and when the position of the non-display area meets a criterion. 10. In the management method according to 8. or 9., The management device, The management method further comprises determining to issue the notification when the area ratio is equal to or greater than a threshold value. 11. In the management method according to 10., The management device, A management method for determining whether or not to change the threshold value for the display shelf at a predetermined position and to issue the notification depending on a time period. 12. In the management method according to any one of 8. to 11., The management device further comprises: Identifying a position of the non-display area from the captured image; the management method determining, based on the position of the non-display area, that the notification is to be made unconditionally if the product is not displayed in a specific position on the display shelf. 13. In the management method according to any one of 8. to 12., The management device further comprises: Identifying a position of the non-display area from the captured image; Identifying a commodity displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; A management method in which text information and an image relating to the identified product are displayed. 14. In the management method according to any one of 8. to 13., The management device further comprises: Identifying a position of the non-display area from the captured image; Identifying a commodity displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; A management method, comprising: calculating a quantity of the identified products based on the area of ​​the non-display area.

[0101] 15. On the computer: A step of obtaining an area of ​​a displayable area on a shelf where products can be displayed; A step of calculating an area ratio of a non-display area in which the products are not displayed, detected from a captured image of the display shelf, to an area of ​​the display shelf; a step of determining whether or not to issue a notification depending on the area ratio; a program for executing a procedure for giving the notification when it is determined that the notification should be given; 16. In the program according to 15., A step of identifying a position of the non-display area from the captured image; The program further causes the computer to execute a procedure of determining that the notification should be made when the area ratio meets the criterion for making the notification and when the position of the non-display area meets the criterion. 17. In the program according to 15. or 16., The program further causes the computer to execute a procedure of determining to issue the notification when the area ratio is equal to or greater than a threshold value. 18. In the program according to 17., The program further causes the computer to execute a procedure for determining whether or not to change the threshold value for the display shelf at a predetermined position and to issue the notification depending on a time period. 19. In the program according to any one of 15. to 18., A step of identifying a position of the non-display area from the captured image; The program further causes the computer to execute a procedure of determining, based on the position of the non-display area, that the notification should be made unconditionally if the product is not displayed in a specific position on the display shelf. 20. In the program according to any one of 15. to 19., A step of identifying a position of the non-display area from the captured image; a step of identifying a commodity that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The program further causes the computer to execute a procedure for displaying text information and an image relating to the identified product. 21. In the program according to any one of 15. to 20., A step of identifying a position of the non-display area from the captured image; a step of identifying a commodity that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The program further causes the computer to execute a procedure of calculating the quantity of the identified products based on the area of ​​the non-display area.

[0102] This application claims priority based on Japanese Patent Application No. 2019-200868, filed November 5, 2019, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0103] 1 Store System 3. Communication Network 5. Camera 10 display shelves 12 Outer Frame 14 Shelves 16 items 20 User terminal 30 Image processing device 100 Management device 102 Area calculation part 104 Judgment section 106 Notification Department 110 Storage device 120 Location identification part 122 Product Specification Department 124 Quantity Calculation Department 130 Display processing unit 132 Display device 140 Order Processing Unit 200, 210, 212, 230, 240 Image information 302 Shelf area information 310 Notification criteria information 320 Display status information 330 Notification area standard information 332 Notification priority information 340 Notification Standard Setting Information 350 Product display information 352 Product information 354 Stock Information 400 notification screen 402 Shelf List Tab 410 Notification List Tab 412 thumbnail images 414 Notification content display area 416 Notification date and time display section 420 Detailed content screen 422 Notification date and time display section 424 Shelf image display unit 426 Confirm button 428 Button 500 Shelf Image Area 1000 Computers 1010 Bus 1020 Processor 1030 Memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface

Claims

1. an area calculation means for acquiring an area of ​​a displayable area in which products can be displayed on a display shelf, and calculating an area ratio of an area of ​​a non-display area in which the products are not displayed, detected from a captured image of the display shelf, to the area of ​​the display shelf; a determination means for determining whether or not to issue a notification in accordance with the area ratio; a notification means for performing the notification when it is determined that the notification should be performed; a position specifying means for specifying a position of the non-display area from the captured image, The determination means determines that the notification is to be performed when the area ratio meets a criterion for performing the notification and the position of the non-display area meets a criterion; a priority order of the notification is set for each of a plurality of areas obtained by dividing an area of ​​one shelf of the display shelf; A shape in which the areas having the same priority order occupy the shelf area is stored in advance in a storage means; The position specifying means specifies a shape of the non-display area from the captured image, The determination means compares the shape of the non-display area identified by the position identification means with the shape stored in the memory means, and the degree of similarity is a predetermined percentage or greater as the criterion for the notification.

2. 2. The management device according to claim 1, The determination means determines that the notification should be performed when the area ratio is equal to or greater than a threshold value.

3. 3. The management device according to claim 2, The determination means determines whether or not to change the threshold value for the display shelf at a predetermined position and to issue the notification depending on a time period.

4. 4. The management device according to claim 1, The determination means determines, based on the position of the non-display area, that the notification should be made unconditionally if the product is not displayed in a specific position on the display shelf.

5. 5. The management device according to claim 1, A management device in which a priority order of the notification is determined for each of a plurality of areas obtained by dividing an area of ​​one shelf of the display shelf, and a different criterion is set for each priority order.

6. 6. The management device according to claim 1, a product identification means for identifying a product that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The management device further comprises a display processing means for displaying text information and an image relating to the identified product.

7. 7. The management device according to claim 1, a product identification means for identifying a product that was displayed in the non-display area of ​​the display shelf based on the position of the non-display area of ​​the display shelf; The management device further comprises a quantity calculation means for calculating a quantity of the identified products based on the area of ​​the non-display area.

8. The management device: Obtaining the area of ​​a displayable area in which the products on the display shelf can be displayed; calculating an area ratio of a non-display area in which the product is not displayed, detected from the captured image of the display shelf, to the area of ​​the display shelf; Identifying a position of the non-display area from the captured image; When determining whether or not to notify based on the area ratio, determining that the notification is to be made when the area ratio satisfies a criterion for making a notification and when the position of the non-display area satisfies a criterion; When it is determined that the notification should be made, the notification is made; a priority order of the notification is set for each of a plurality of areas obtained by dividing an area of ​​one shelf of the display shelf; The shape of the shelf area occupied by the areas with the same priority is stored in advance in a storage means, The management device, When identifying the position, a shape of the non-display area is identified from the captured image; When making the judgment, the shape of the identified non-display area is compared with the shape stored in the memory means, and the degree of similarity being a predetermined percentage or higher is set as the criterion for the notification.

9. On the computer, A step of obtaining an area of ​​a displayable area on a shelf where products can be displayed; A step of calculating an area ratio of a non-display area in which the products are not displayed, detected from a captured image of the display shelf, to an area of ​​the display shelf; a step of determining whether or not to issue a notification depending on the area ratio; a step of performing the notification when it is determined that the notification should be performed; a step of identifying a position of the non-display area from the captured image; a step of determining that the notification is to be made when the area ratio satisfies a criterion for making the notification and the position of the non-display area satisfies a criterion in the step of determining; a priority order of the notification is set for each of a plurality of areas obtained by dividing an area of ​​one shelf of the display shelf; The shape of the shelf area occupied by the areas with the same priority is stored in advance in a storage means, In the step of identifying the position, a shape of the non-display area is identified from the captured image; In the determination step, the program compares the shape of the identified non-display area with the shape stored in the memory means, and sets the degree of similarity to a predetermined percentage or higher as the criterion for the notification.

Images