Image processing system, image processing device, image processing method and program

Abstract

An image processing system, an image processing device, an image processing method, and a program are provided that can easily and accurately detect a shortage of a product. [Solution] The image processing system 1 includes an extraction unit 11b that extracts product areas corresponding to each product from an image of a product display shelf, a setting unit 11a that sets product display areas for each product type in the image, and a determination unit 11c that determines whether a product is in short supply for each product type based on the correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area.

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an image processing device, an image processing method, and a program for detecting a shortage of a product. [Background technology]

[0002] In a store, the display status of each product on the product shelves is managed. When the number of products displayed on the product shelves decreases or when an item is out of stock, a store clerk replenishes the product. A product shortage can be detected, for example, based on captured images of the product shelves.

[0003] The following Patent Document 1 describes an image processing device for detecting shortages of products. This image processing device acquires an image of a display area where products are displayed, calculates the area of ​​the area that would appear if the product were removed from the image, determines whether the product is in short supply based on the calculated area, and notifies a store clerk of the determined shortage via an information terminal. This allows the store clerk to smoothly replenish the products that are in short supply. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-053849 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 requires extracting from the captured image the non-display area that appears when the product has been removed (for example, the rear wall surface, etc.). However, the state of the non-display area differs from store to store. Therefore, in order to accurately extract the non-display area, learning and tuning must be performed for each store. This process requires a great deal of time and effort.

[0006] In view of the above, an object of the present invention is to provide an image processing system, an image processing device, an image processing method, and a program that can easily and accurately detect a shortage of a product. [Means for solving the problem]

[0007] A first aspect of the present invention relates to an image processing system, which includes a setting unit for setting product display areas for each product type in a captured image of a product display shelf, an extraction unit for extracting product areas corresponding to each product from the captured image, and a determination unit for determining whether a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area.

[0008] As products are removed from the product shelves, the total area of ​​the product areas included in the display area decreases, and the correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area changes. Therefore, this correlation value can accurately determine the product shortage status for that display area. Furthermore, unlike background images, the product images themselves do not differ from store to store, so product areas can be easily extracted from captured images using a general-purpose product recognition algorithm without the need for learning or tuning for each store. Therefore, the image processing system according to this embodiment can easily and accurately detect product shortages.

[0009] In the image processing system according to this aspect, the correlation value may be a ratio of the total area to the area of ​​the display area.

[0010] As described above, as products are removed from the product display shelves, the total area of ​​the product areas included in the display area decreases, and the ratio of the total area to the display area area decreases. Therefore, by using this ratio as a correlation value, it is possible to accurately detect a shortage of products.

[0011] In the image processing system according to this aspect, the judgment unit may be configured to set a criterion for judging the shortage state for each of the product types based on the size of the product area for each of the product types on the captured image.

[0012] The larger the size of the product, the fewer the number of products included in the total product area. Therefore, when determining a product shortage by comparing the above correlation value with a fixed determination criterion, the number of products remaining for each product type when it is determined that a shortage has occurred may differ depending on the size of the products for each product type. In this case, for example, for a product type with individually small products, there will be more products remaining when it is determined that a shortage has occurred, and for a product type with individually large products, there will be fewer products remaining when it is determined that a shortage has occurred.

[0013] In contrast, with the above configuration, the criteria for determining whether a product type is in short supply are set based on the size of the product area for each product type in the captured image. Therefore, the number of remaining products for each product type when determining whether a product is in short supply can be made approximately the same. Therefore, when the result of the detection of a shortage is notified to a user, such as a store clerk, the notification is made to the user when the remaining number for each product type reaches approximately the same number. This allows the user to properly and smoothly replenish the products.

[0014] In this case, the judgment unit may be configured to set the judgment criteria for each of the product types based on the correlation between the size of the product area of ​​each of the product types on the captured image and the size of the display area.

[0015] The proportion of a single product area relative to the display area varies depending on the size of the display area. Therefore, the larger the display area, the greater the number of remaining products for the same correlation value. Therefore, if the judgment criteria were set based solely on the size of the product area, the number of remaining products when it is determined that a shortage has occurred may vary depending on the size of the display area. In contrast, the above configuration sets the judgment criteria based on the correlation between the size of the product area and the size of the display area. Therefore, the number of remaining products when it is determined that a shortage has occurred can be made similar between product types, allowing users such as store clerks to be notified of the shortage at the appropriate time.

[0016] In this configuration, the size of the product region used to set the determination criterion may be the surface area of ​​the product region.

[0017] According to this configuration, the determination criteria can be smoothly set based on the area of ​​the product region, which reflects the size of the entire product region.

[0018] In the image processing system according to this aspect, the setting unit may be configured to extract a product group area in which products of the same product type are lined up consecutively from the product area extracted from the captured image, and to set the display area for that product type based on the extracted product group area.

[0019] According to this configuration, the setting unit automatically extracts a product group area for each product type from the captured image, so that the extracted product group area can be set as is as the display area for the corresponding product type, or the display area for the corresponding product type can be easily set by accepting fine adjustments to the product group area from the administrator.

[0020] In this case, the setting unit may be configured to extract, as the product group area, a plurality of product areas that are identical or similar in at least one of area, height, width, height-to-width ratio, and shape and that are arranged consecutively from the product areas extracted from the captured image.

[0021] According to this configuration, it is possible to properly identify a plurality of product areas corresponding to products of the same product type. Therefore, it is possible to properly extract a product group area from a captured image and properly set a display area. can.

[0022] The image processing system according to this aspect may include a notification processing unit that notifies a user's mobile terminal of the result of the shortage determination made by the determination unit.

[0023] According to this configuration, it is possible to notify users, such as store staff, of a shortage of a product, thereby enabling the users to smoothly replenish the corresponding product display shelves with the product that is running low.

[0024] A second aspect of the present invention relates to an image processing device, which includes a setting unit for setting product display areas for each product type in a captured image of a product display shelf, an extraction unit for extracting product areas corresponding to each product from the captured image, and a determination unit for determining whether a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area.

[0025] According to the image processing device of this aspect, the same effects as those of the first aspect can be achieved.

[0026] A third aspect of the present invention relates to an image processing method, which includes a step of setting a product display area for each product type in a captured image of a product display shelf, a step of extracting a product area corresponding to each product from the captured image, and a step of determining whether or not a product is in short supply for each product type based on a correlation value between a total area of ​​the product areas included in the display area and an area of ​​the display area.

[0027] According to the image processing method of this aspect, the same effects as those of the first aspect can be achieved.

[0028] A fourth aspect of the present invention relates to a program for causing a computer to execute predetermined functions, which causes the computer to execute a function for setting a product display area for each product type in a captured image of a product display shelf, a function for extracting a product area corresponding to each product from the captured image, and a function for determining whether a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area.

[0029] According to the program of this aspect, the same effects as those of the first aspect can be achieved. [Effects of the Invention]

[0030] As described above, according to the present invention, it is possible to provide an image processing system, an image processing device, an image processing method, and a program that are capable of easily and accurately detecting a shortage of a product.

[0031] The effects and significance of the present invention will become more apparent from the following description of the embodiments, however, the embodiments shown below are merely examples of how the present invention can be implemented, and the present invention is not limited to the embodiments described below. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a diagram showing the configuration of an image processing system according to an embodiment. [Figure 2] FIG. 2 is a plan view schematically illustrating an example of a store layout according to the embodiment. [Figure 3] FIG. 3 is a front view schematically showing the configuration of a product shelf according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of an image processing device and a mobile terminal according to an embodiment. [Figure 5] FIG. 5 is a diagram showing a method for setting a display area for a captured image according to the embodiment. [Figure 6]FIG. 6 is a flowchart showing a process for setting a display area for a captured image according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a part of a display area setting image according to the embodiment. [Figure 8] FIG. 8 is a diagram showing the configuration of display area management information according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating a method for determining whether a shortage of a product has occurred according to an embodiment. [Figure 10] FIG. 10 is a flowchart showing a process for monitoring whether a product shortage has occurred according to an embodiment. [Figure 11] FIG. 11 is a diagram showing the configuration of shortage status management information according to the embodiment. [Figure 12] 12(a) and 12(b) are diagrams showing notification screens displayed on the mobile terminal of a store clerk in response to a shortage notification according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing a process for transmitting a shortage unresolved notification executed by the control unit of the image processing apparatus according to the embodiment. [Figure 14] 14A and 14B are diagrams showing notification screens according to the embodiment that are displayed on the mobile terminal of the manager upon receiving a notification that the shortage has not been resolved. [Figure 15] Fig. 15(a) is a flowchart showing a process for setting criteria for determining a shortage state, which is executed by a control unit (determination unit) of an image processing device, according to Modification Example 1. Fig. 15(b) is a flowchart showing the process in step S403 of Fig. 15(a) according to Modification Example 1. [Figure 16] FIG. 16 is a flowchart showing a process for monitoring whether or not a shortage of a product has occurred, according to the second modification. DETAILED DESCRIPTION OF THE INVENTION

[0033] FIG. 1 is a diagram showing the configuration of an image processing system 1 according to an embodiment.

[0034] The image processing system 1 includes an image processing device 10 and an edge PC (edge ​​computing) 20. The image processing device 10 and the edge PC 20 can communicate with each other via a network N1. The edge PC 20 adds additional information for identifying the camera 30 to an image captured by a camera 30 and transmits the image to the image processing device 10. In FIG. 1, one camera 30 is connected to the edge PC 20, but multiple cameras 30 can be connected to one edge PC 20. For example, six cameras 30 are connected to one edge PC 20.

[0035] The edge PC 20, the camera 30, and the transmitting / receiving device 40 are installed in a store where the product shelves 2 are installed. The edge PC 20 may be installed on the top surface of the product shelves 2. The image processing device 10 may be located in the store, or may be located outside the store. When the image processing device 10 is installed in the same store where the product shelves 2 are installed, the network N1 is a LAN (Local Area Network) installed in the store. When the image processing device 10 is installed in a facility outside the store, the network N1 is an external communication network such as the Internet.

[0036] Each camera 30 captures an image of a different product shelf 2 from the front. The edge PC 20 adds additional information (identification information) of each camera 30 to the captured image received from that camera 30 and transmits the image to the image processing device 10. When a large number of cameras 30 are installed in a store, multiple edge PCs 20 are installed.

[0037] The image processing device 10 processes the captured image received from the edge PC 20 and determines whether or not the product displayed on the corresponding shelf 2 is in short supply for each product type. When a product of a predetermined product type is in short supply, the image processing device 10 transmits a notification to the mobile terminal 50 carried by the store clerk 3 via the transmission / reception device 40. The image processing device 10 is capable of communicating with the transmission / reception device 40 via the network N1. The image processing device 10 is, for example, a server computer or a personal computer. The image processing device 10 may also be a dedicated device.

[0038] The transmitting / receiving device 40 is a communication device for wirelessly communicating with the mobile terminal 50. The transmitting / receiving device 40 is installed on the ceiling of the store, the top surface of the product shelf 2, or the like. The transmitting / receiving device 40 may be a wireless LAN router. The mobile terminal 50 is, for example, a mobile phone such as a smartphone. The mobile terminal 50 is distributed to each store clerk 3 and carried by the corresponding store clerk 3. The mobile terminal 50 is also distributed to the manager of the store.

[0039] FIG. 2 is a plan view schematically showing an example of the layout of the store S1.

[0040] Here, the store S1 is divided into a sales floor S11, a storage room S12, and a checkout area S13. A plurality of product shelves 2 are installed in the sales floor S11, and a plurality of cameras 30 are installed facing each of the product shelves 2. Here, a camera 30 is assigned to each product shelf 2. Each camera 30 captures an image of the entire front of the corresponding product shelf 2. However, this is not limited to this, and a single camera 30 may be assigned to a plurality of product shelves 2, or a single camera 30 may capture an image of only a portion of the product display shelf 201 of a single product shelf 2.

[0041] The sales floor S11 has an entrance S14. In the storage room S12, products for replenishment are stocked. The checkout area S13 is separated from the sales floor S11 by a counter 4. Two cash registers 5 are installed at the counter 4. In the example of Figure 2, four store clerks 3 are on duty at the store S1. Two of the store clerks 3 are in charge of their respective cash registers 5. The remaining two store clerks 3 are in charge of the sales floor. One of the store clerks 3 is in the storage room S12 to replenish products. The storage room S12 can be entered through the store clerk-only entrance S15. As described above, each store clerk 3 carries a mobile terminal 50.

[0042] FIG. 3 is a front view schematically showing the configuration of the product shelf 2. As shown in FIG.

[0043] The product shelf 2 is equipped with a plurality of product display shelves 201 for displaying products. Each product display shelf 201 displays products of different product types, one for each product type. As a result, the product display shelf 201 is divided into a plurality of product type areas 202 arranged in the left-right direction. A label 203 indicating the position of each product display shelf 201 on the product shelf 2 is attached to the right side of each product display shelf 201. In addition, a label 204 indicating the identification symbol of the product shelf 2 is attached to the upper left corner of the product shelf 2. The identification symbol of the product shelf 2 is, for example, a number assigned consecutively to all product shelves 2 in the store S1.

[0044] Fig. 4 is a block diagram showing the configurations of the image processing device 10 and the mobile terminal 50. For convenience, two mobile terminals 50 are shown in Fig. 4, but in reality, there are as many mobile terminals 50 as there are store clerks and managers working in the store S1.

[0045] The image processing device 10 includes a control unit 11, a storage unit 12, a display unit 13, an input unit 14, and a communication unit 15.

[0046] The control unit 11 includes an arithmetic processing circuit such as a CPU (Central Processing Unit) and controls each unit according to a program stored in the storage unit 12. The storage unit 12 includes memories such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk, and stores the program. The storage unit 12 is also used as a work area when the control unit 11 executes the program.

[0047] The display unit 13 includes a display means such as a liquid crystal monitor, and displays a predetermined screen under the control of the control unit 11. The input unit 14 includes an input means such as a mouse or keyboard, and outputs information according to the input to the control unit 11. The communication unit 15 communicates with the edge PC 20 and the mobile terminal 50 under the control of the control unit 11. As described above, communication with the mobile terminal 50 is performed via the transmission / reception device 40.

[0048] In this embodiment, a program for determining the shortage status of a product based on an image captured by the camera 30 is stored in the storage unit 12. This program causes the control unit 11 to execute the functions of a setting unit 11a, an extraction unit 11b, a determination unit 11c, and a notification processing unit 11d.

[0049] The setting unit 11a executes a process for setting a display area for each product type in the captured image. A display area is an area in the captured image where products of the same product type are lined up consecutively. The process for setting the display area by the setting unit 11a will be described later with reference to FIGS. 5 and 6.

[0050] The extraction unit 11b extracts a product area corresponding to each product from the captured image captured by the camera 30. The extraction unit 11b executes an algorithm for extracting the product area. This algorithm is a general-purpose algorithm that can extract a product area corresponding to a product from the captured image through a learning process using images of various existing products. The extracted product area is a rectangular area that surrounds the image of the product.

[0051] The determination unit 11c determines whether or not a product is in short supply (whether or not a product needs to be replenished) for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area. The process of determining whether or not a product is in short supply by the determination unit 11c will be described later with reference to FIGS. 9 and 10.

[0052] The notification processing unit 11d executes a process of notifying the mobile terminal 50 of the store clerk 3 of the result of the shortage state determination made by the determination unit 11c. As described above, this notification (shortage notification) is made via the transmitting / receiving device 40. The notification process by the notification processing unit 11d will be described later with reference to Figs. 10 to 14(b).

[0053] The storage unit 12 also stores clerk management information that links together information about each clerk 3 (such as name), identification information about the mobile terminal 50 carried by each clerk 3, and communication information for communicating with the mobile terminal 50. The storage unit 12 also stores manager management information that links together information about the store manager (such as name), identification information about the mobile terminal 50 carried by the manager, and communication information for communicating with the mobile terminal 50.

[0054] Furthermore, the storage unit 12 stores display area management information shown in Fig. 8 and shortage status management information shown in Fig. 11. These pieces of information will be described later with reference to Figs. 8 and 11.

[0055] The mobile terminal 50 includes a control unit 51, a storage unit 52, a display / input unit 53, a speaker 54, and a communication unit 55. As described above, the mobile terminal 50 is a mobile phone such as a smartphone. The mobile terminal 50 may also be another mobile terminal such as a tablet.

[0056] The control unit 51 includes a processing circuit such as a CPU, and controls each unit according to a program stored in the storage unit 52. The storage unit 52 includes memories such as a ROM and a RAM, and stores the program. The storage unit 52 is also used as a work area when the control unit 51 executes processing.

[0057] An application program for managing the shortage status of a product is stored in the storage unit 52. In accordance with this application program, the control unit 51 receives the shortage notification from the image processing device 10 and causes the display input unit 53 to display a notification screen for informing the user of the shortage status.

[0058] The display / input unit 53 is configured by overlaying a touch panel on a liquid crystal display. A display unit consisting of a liquid crystal display and an input unit consisting of a group of operation keys may be separately arranged instead of the display / input unit 53. The speaker 54 outputs a predetermined sound in response to control from the control unit 51.

[0059] FIG. 5 is a diagram showing a method for setting the display area R13 for the captured image P1.

[0060] FIG. 5 shows a shelf image P11 of a portion of one product display shelf 201 from a captured image P1 acquired by one camera 30. The shelf image P11 includes product images P12 of three product types. These product images P12 are product images related to beverages. In other words, beverages of different product types are displayed on the product display shelf 201 of the product shelf 2 corresponding to the shelf image P11. A rectangular area surrounding each product image P12 is extracted as a product area R11.

[0061] The display area R13 is set as a rectangular area surrounding a group of product areas R11 for each product type. That is, first, the extraction unit 11b in Fig. 4 extracts product areas R11 corresponding to each product from the shelf image P11 using the above-mentioned algorithm. Next, the setting unit 11a in Fig. 4 extracts product group areas R12 in which products of the same product type are lined up consecutively from the extracted product areas R11, and sets the display area R13 for each product type based on the extracted product group areas R12.

[0062] Here, the setting unit 11a extracts predetermined parameters capable of identifying products of the same type from the product area R11 and uses the extracted parameters to distinguish the product areas R11 of the same product type. For example, the setting unit 11a acquires the area (e.g., the number of pixels) and shape of the product area R11 as the parameters.

[0063] However, without being limited to this, other parameters capable of identifying similar products may be extracted from the product region R11. The parameters capable of identifying similar products may be at least one of area, height, width, height-to-width ratio, and shape. For example, the setting unit 11a may use the height-to-width ratio of the product region R11 as the parameter. Alternatively, the setting unit 11a may use the height and width of the product region R11 as the parameter.

[0064] The setting unit 11a extracts, as the product group area R12, multiple product areas R11 that have the same or similar parameters and are arranged continuously in the vertical and horizontal directions. For example, if the parameters are the area and shape of the product area R11, the setting unit 11a extracts, as the product group area R12, multiple product areas R11 that have the same or similar areas and shapes and are arranged continuously in the vertical and horizontal directions.

[0065] Here, the range of approximation of the parameters is the allowable range of deviation between the parameters that is allowed for product regions R11 of the same type of product. For example, if products of the same type are displayed shifted forward or backward, the area and shape of the product regions R11 of these products may differ slightly. Furthermore, even if products of the same type are lined up neatly side by side or up and down, the area and shape of the product regions R11 may differ slightly due to the angle of view and imaging direction of the camera 30. The above-mentioned range of approximation is set so that these differences can be treated as product regions R11 of the same type of product.

[0066] The setting unit 11a sets a display area R13 for each product type based on the extracted product group area R12. Here, the smallest area that surrounds each product group area R12 in a rectangle is set as the display area R13.

[0067] Note that the method for setting the display area R13 based on the product group area R12 is not limited to this. For example, a rectangular area that surrounds the product group area R12 and extends up and down to the upper and lower boundaries of the shelf image P11 may be set as the display area R13. Alternatively, the display areas R13 may be set so that the boundaries of adjacent display areas R13 are in contact, or the product group area R12 may be set as the display area R13 as is.

[0068] 5, the position and range of each display area R13 is defined by coordinate values ​​on the X and Y axes, with the upper left corner of the captured image P1 (entire area) as the origin. Specifically, as shown for the leftmost display area R13, the position and range of the display area R13 on the captured image P1 are defined by the X and Y coordinate values ​​(Xmax, Ymax, Xmin, Ymin) of the four sides of the display area R13. The positions and ranges of the other display areas R13 in the captured image P1 are defined in a similar manner.

[0069] FIG. 6 is a flowchart showing the process of setting the display area R13 for the captured image P1.

[0070] The processing in Fig. 6 is executed when a user such as an administrator selects a setting mode for the display area R13 on a setting screen displayed on the display unit 13 of the image processing device 10. Fig. 6 shows processing for a captured image P1 from one camera 30. The control unit 11 executes the processing in Fig. 6 for each camera 30. The processing in Fig. 6 is performed for all product images P12 included in the captured image P1.

[0071] In the flowchart of Fig. 6, the process of step S103 is performed by the function of the extraction unit 11b, and the processes of steps S104 and S105 are performed by the function of the setting unit 11a. In the following, the description will be made on the assumption that the control unit 11 executes the process of Fig. 6 using these functions.

[0072] When the setting mode for the display area R13 is started (S101: YES), the control unit 11 acquires the captured image P1 from the camera 30 (S102). As described above, the captured image P1 from the camera 30 is transmitted to the control unit 11 via the edge PC 20. At this time, the edge PC 20 adds additional information (identification information) of the camera 30 to the captured image P1 and transmits it. As a result, the control unit 11 executes the flowchart of FIG. 6 for each camera 30.

[0073] The control unit 11 extracts a product area R11 from the captured image P1 acquired from the camera 30 via the edge PC 20 (S103), and further extracts a product group area R12 in which product areas R11 of the same product type are lined up consecutively (S104). The control unit 11 sets a rectangular area surrounding the extracted product group area R12 as the display area R13 (S105). The processing of steps S102 to S105 is the same as the processing described with reference to FIG.

[0074] The control unit 11 causes the display unit 13 to display an input screen including a display area setting image in which the display areas R13 for each product type set in step S105 are superimposed on the captured image P1 of the entire product shelf 2 (S106). The display area setting image is an image for accepting corrections to each display area R13 from the manager. The control unit 11 accepts corrections to the display areas shown in the display area setting image from a user such as the manager (S107).

[0075] FIG. 7 is a diagram showing a part of the display area setting image P2.

[0076] The display area setting image P2 includes a product image P21 and a frame P22 that indicates the display area R13. A user such as an administrator appropriately operates the input unit 14 (for example, a mouse) to modify the rectangular shape of the frame P22 (display area R13). For example, the administrator modifies the frame P22 (display area R13) by grabbing and moving a corner or side of the frame P22 with the mouse. In response to this modification, the X and Y coordinate values ​​(Xmax, Ymax, Xmin, Ymin) that define the position and range of the display area R13 are changed. The other display areas R13 can also be modified in a similar manner.

[0077] Returning to FIG. 6, the control unit 11 accepts input of product information for each display area R13 via the above-mentioned input screen (S108). For example, the manager clicks on one of the frames P22 on the display area setting image P2 and inputs the product category and product type (e.g., product name) of the product to be displayed in the display area R13 corresponding to that frame P22 into the input fields on the input screen. The manager inputs this product information for all of the frames P22 (display areas R13) on the display area setting image P2. The user then operates the confirm button included in the input screen.

[0078] When the input is completed in this way, the control unit 11 registers the information according to the input in the display area management information (see FIG. 8) (S109), whereby the control unit 11 ends the processing of FIG.

[0079] 6 (display area setting mode) is preferably executed in a state where products are neatly displayed without any omissions in each product type area 202 of the product display shelf 201. This makes it easier for each display area R13 automatically set by the setting unit 11a to match the original display area R13 for the same product group. This reduces the need for the user to modify the display area R13 via the display area setting image P2.

[0080] Furthermore, when the display area R13 and part of the product information are changed due to the display of new products, etc., instead of executing the process of FIG. 6 again, an input screen may be displayed on which the display area R13 and product information set in the previous process can be modified. In this case, the input screen displays the display area R13 set in the previous process in the display area setting image P2, and displays a list of product information corresponding to each display area R13. The administrator only needs to modify the area and change the product information for the display area R13 whose product type has changed since the previous process. This can reduce the burden on the user.

[0081] FIG. 8 is a diagram showing the configuration of the display area management information.

[0082] 8, the display area management information is managed for each camera 30. The display area management information is configured by linking together a display area number, display area information, and product information.

[0083] The "display area number" is a number automatically assigned to each display area R13. The "display area information" is information that defines the position and range of the display area R13 on the captured image P1, and is defined by the X and Y coordinate values ​​(Xmax, Ymax, Xmin, Ymin) on the captured image, as described above. When the display area R13 is modified in step S107 of FIG. 6, the X and Y coordinate values ​​for the modified display area R13 are registered in the "display area information."

[0084] "Product information" is the product information entered in step S104 of FIG. 6 and is composed of "product category" and "product type." Here, the product name is registered as the "product type." However, this is not limited to this, and the "product type" may be other information that can identify the product type, such as a product code. Furthermore, the "product type" may include multiple items, such as the product name and product code.

[0085] FIG. 9 is a diagram showing a method for determining whether a shortage of a product has occurred.

[0086] In this embodiment, the shortage status of a product in each product type area 202 is determined based on the display area R13 and the product area R11 in the captured image P1. The display area R13 in the captured image P1 corresponds to the product type area 202 of the product display shelf 201. The control unit 11 (determination unit 11c) of the image processing device 10 determines whether a product shortage has occurred in each product type area 202, using the display area R13 in the captured image P1 as the target. The control unit 11 manages the product type area 202 in which a product shortage has occurred by the display area number assigned to the display area R13 in the captured image P1. This management is performed using shortage status management information (FIG. 11), which will be described later. The control unit 11 also manages the product types in which a product shortage has occurred by linking the display area number registered in the shortage status management information (FIG. 11) with the display area number in the display area management information (FIG. 8).

[0087] In this embodiment, whether a product is in short supply is determined for each product type based on the correlation value between the total area of ​​the product areas R11 included in each display area R13 and the area of ​​each display area R13. This determination is made by the function of the determination unit 11c in FIG. 4.

[0088] Compared to Fig. 7, in Fig. 9, the area of ​​the product area R11 included in each display area R13 has decreased due to product purchases. Therefore, the above-mentioned correlation value changes depending on the number of products remaining in the product type area 202. Therefore, by comparing this correlation value with a predetermined correlation value, it is possible to determine the shortage of products in each product type area 202 (product type).

[0089] For example, the correlation value can be calculated by the following equation (1).

[0090] Correlation value = (total area of ​​product areas included in the display area) / (area of ​​the display area) ... (1)

[0091] In this case, the correlation value decreases as the number of products remaining in the product type area 202 decreases. The determination criterion is a threshold value applied to the correlation value. If the correlation value is equal to or less than this threshold value, it is determined that a product shortage has occurred for the product type corresponding to the product type area 202. The threshold value may be set as a default, or may be arbitrarily set by the administrator depending on the desired remaining quantity for product replenishment. A threshold value may also be set for each display area R13.

[0092] When the correlation value exceeds the threshold, there may be cases where no products remain in the product type area 202 due to bulk buying or the like. In this case, the correlation value becomes zero, and the correlation value becomes equal to or less than the threshold, so it is determined that a shortage has occurred. At this time, it may be determined that a stockout has occurred as the content of the shortage. Also, even if there is no bulk buying, if the shortage has occurred (correlation value = 0) while a response to the shortage is delayed, it may be determined that a stockout has occurred as the content of the shortage.

[0093] The correlation value is not limited to the ratio of the above formula (1). For example, the correlation value may be calculated using any of the following formulas.

[0094] Correlation value = (area of ​​display area) / (total area of ​​product areas included in the display area) ... (2) Correlation value = (area of ​​display area - total area of ​​product areas included in the display area) / (area of ​​display area) ... (3)

[0095] In the formulas (2) and (3), the correlation value increases in accordance with the decrease in the number of products remaining in the product type area 202. Therefore, in these correlation values, when the correlation value is equal to or greater than a threshold value, it is determined that a product shortage has occurred in the product type corresponding to the product type area 202.

[0096] For convenience, Fig. 9 shows a shelf image P11 obtained when the product display shelf 201 is imaged from directly in front, but in many cases, the product display shelf 201 is imaged by the camera 30 from above. Furthermore, the product display shelf 201 often has multiple products lined up and down, left and right, as well as front and back. In these situations, when a product in the front row is removed, the camera 30 images the products in the second row, and when the product in the second row is removed, the camera 30 images the products in the third row. In this case, the products in the second and third rows are located at the back, so the product area R11 becomes smaller.

[0097] The correlation value described above is not the number of product areas R11 included in a display area R13 in the captured image P1, but a value that defines the correlation between the total area of ​​the product areas R11 included in that display area R13 and the area of ​​that display area R13. Therefore, even when products are removed in order from the front, the correlation value changes depending on the removal of products. Therefore, the shortage status of each product type can be properly determined from the correlation value.

[0098] In the above correlation value calculation formula, the total area may be calculated as the sum of the areas (e.g., the number of pixels) of the product areas R11 that are substantially included in each display area R13. For example, depending on the display state of the products, a product area R11 may protrude from the display area R13. In this case, if a certain percentage (e.g., 80%) of this product area R11 is included in the display area R13, the area of ​​this product area R11 is added to the total area for that display area R13; otherwise, the area of ​​this product area R11 is not added to the total area.

[0099] FIG. 10 is a flowchart showing the process of monitoring whether or not a product shortage has occurred.

[0100] 10 shows processing for a captured image P1 from one camera 30. The control unit 11 executes the processing of FIG. 10 for each camera 30. As in the case of FIG. 6, the camera 30 is identified based on additional information (identification information of the camera 30) that the edge PC 20 adds to the captured image P1 when transmitting the captured image P1 to the control unit 11. The processing of FIG. 10 is performed for all display areas R13 included in the captured image P1.

[0101] In the flowchart of Fig. 10, the process of step S202 is performed by the function of extraction unit 11b, the processes of steps S203 to S206 are performed by the function of determination unit 11c, and the process of step S209 is performed by the function of notification processing unit 11d. In the following, the description will be made assuming that control unit 11 executes the process of Fig. 10 using these functions.

[0102] The control unit 11 acquires a captured image P1 from the camera 30 via the edge PC 20 (S201), and extracts a product area R11 from the acquired captured image P1 (S202). Next, the control unit 11 calculates the total area of ​​the product areas R11 for each display area R13 in the captured image P1 (S203), and calculates a correlation value between the calculated total area and the area of ​​the display area (S204). The control unit 11 then compares the correlation value of each display area R13 with a determination criterion (S205), and determines whether or not there is a product type area 202 (product type) where products are in short supply (S206). The processing of steps S202 to S205 is the same as the processing described with reference to FIG. 9.

[0103] If the determination in step S206 is NO, the control unit 11 ends the processing in Fig. 10 and proceeds again from step S201. On the other hand, if the determination in step S206 is YES, the control unit 11 determines whether a new product type region 202 in which a product is in short supply has occurred (S207).

[0104] Specifically, the control unit 11 determines whether any of the display area numbers (see FIG. 8) corresponding to the product type areas 202 in a shortage state detected in steps S205 and S206 are registered in the shortage state management information (see FIG. 11) that manages the display area R13 in a shortage state. If any of the display area numbers is not registered in the shortage state management information, the control unit 11 determines YES in step S207, and if both display area numbers are registered in the shortage state management information, the control unit 11 determines NO in step S207.

[0105] If the determination in step S207 is NO, the control unit 11 ends the processing of FIG. 10 and resumes the processing from step S201. On the other hand, if the determination in S207 is YES, the control unit 11 sets the product type area 202 (product type) corresponding to the display area number that was not registered in the shortage status management information as the product type area 202 (product type) where a new shortage has occurred, and registers information about the product type area 202 in the shortage status management information (S208). Then, the control unit 11 transmits a shortage notification to the mobile terminal 50 carried by the store clerk 3 (S209). As described above, this transmission is performed via the transmission / reception device 40.

[0106] FIG. 11 is a diagram showing the configuration of the shortage status management information.

[0107] 11, shortage status management information is managed for each camera 30. The shortage status management information is configured by linking together the display area number, the date and time when the shortage occurred, the person who responded, and the response status.

[0108] As described above, when the control unit 11 determines that a new product type region 202 in a shortage state has been created in step S207 of Fig. 10, the control unit 11 adds one row of information to the shortage state management information. Furthermore, when the control unit 11 receives a completion notification from the mobile terminal 50 that the product type region 202 that was in a shortage state has been restocked with products, the control unit 11 deletes the row of information corresponding to the product type region 202 from the shortage state management information.

[0109] The "display area number" is registered as the "display area number" of the display area R13 in the captured image P1 that corresponds to the product type area 202 determined to have a shortage in step S206 of FIG. 10, among the "display area numbers" in the display area management information of FIG. 6. The "date and time of shortage occurrence" is registered as the date and time when each row was added to the shortage status management information. The "responder" is registered as information indicating the store clerk 3 who will replenish the corresponding product type area 202. The "responder status" is registered as information indicating the progress of replenishment of the corresponding product type area 202. The "responder" and "response status" are registered in response to a notification from the mobile terminal 50 carried by the store clerk 3, as described below. Note that if it is further determined that a shortage has occurred as described above, an item for managing whether or not a shortage exists may be further linked to the information in each row.

[0110] 10, the control unit 11 transmits a shortage notification to the mobile terminal 50 of each store clerk 3 for all "display area numbers" managed in the shortage status management information. That is, every time a new product type area 202 (product type) in a shortage state occurs, a shortage notification is transmitted to the mobile terminal 50 of each store clerk 3 for all display area numbers registered in the shortage status management information (FIG. 11), that is, for all product type areas 202 for which product replenishment has not been completed.

[0111] The shortage notification includes notification information linking information linked to each "display area number" in the shortage status management information of FIG. 11 with "product information" linked to these "display area numbers" in the display area management information (see FIG. 8). In this embodiment, the shortage notification is sent to the mobile device 50 via a push notification. However, the method is not limited to this, and the shortage notification may be sent to the mobile device 50 of each store clerk 3 by other methods such as email.

[0112] 12(a) and 12(b) are diagrams showing notification screens displayed on the mobile terminal 50 of the store clerk 3 in response to the shortage notification in step S209 of FIG.

[0113] In response to receiving the push notification, the control unit 51 of the mobile terminal 50 adds a display indicating the arrival of a new push notification to the icon of the application program for managing the shortage status of a product. At this time, the control unit 51 causes the speaker 54 to output a chime sound indicating the arrival of the push notification.

[0114] When the store clerk 3 touches this icon in response to these notifications, a notification item 501 shown in FIG. 12(a) is displayed on the display input unit 53. The notification item 501 includes a display indicating that a new product type region 202 has become out of stock, and a display indicating the total number of product type regions 202 that are out of stock and have not yet been replenished. If it is further determined that a stockout has occurred as described above, the notification item 501 may further include information such as "xx items out of stock," to notify the store clerk that a shortage has occurred. When the store clerk 3 touches the notification item 501, a shortage notification screen 510 shown in FIG. 12(b) is displayed.

[0115] The shortage notification screen 510 displays a plurality of display areas 511 separated by dashed lines and scrollable up and down. Each display area 511 displays one line of information in the shortage status management information (see FIG. 11) and product information in the display area management information linked to that line via the display area number. When the shortage status management information consists of five lines of information as shown in FIG. 11, five display areas 511 are arranged vertically on the shortage notification screen 510. Each display area 511 includes the time when the shortage occurred, the product category and product name of the product in which the shortage occurred, the person in charge of handling the shortage, and a button 512.

[0116] Display items other than button 512 are displayed based on the above-mentioned notification information included in the shortage notification. The display of button 512 in display area 511 corresponding to a row for which a "responder" has already been set in the display area management information of FIG. 8 reflects the content of the "response status" of that row. Button 512 in display area 511 corresponding to a row for which a "responder" has not yet been set displays a message for accepting from store clerk 3 a request to respond to product replenishment. If it is further determined that a stockout state has occurred as described above, the display area for the product that is out of stock may further include information such as "out of stock," thereby notifying the store clerk that a stockout has occurred.

[0117] By scrolling the shortage notification screen 510 up and down, the store clerk 3 can see all product types (product names) for which product replenishment is currently incomplete, and can also see which product types have not yet been replenished by viewing the display of buttons 512. When the store clerk 3 wishes to replenish an incomplete product type, he or she touches button 512 in the display area 511 corresponding to that product type. This causes the control unit 51 of the mobile terminal 50 to send a response notification including the display area number corresponding to the display area 511 to the image processing device 10 together with the store clerk's own identification information.

[0118] The control unit 11 of the image processing device 10 extracts the clerk information of the clerk 3 linked to the received identification information from the clerk management information described above, and registers the extracted clerk information in the "responder" field of the corresponding row in the shortage status management information in FIG. 11. The control unit 11 also changes the "response status" of that row to "responding." Furthermore, the control unit 11 generates new notification information from the shortage management information and display area management information after these items have been updated, and sends a shortage notification including the generated notification information to the mobile terminal 50 of each clerk 3. As a result, the shortage notification screen 510 after the above items have been updated is displayed on each mobile terminal 50.

[0119] Thereafter, when the clerk 3 has completed replenishing the products, he / she executes the above-mentioned application program on his / her mobile terminal 50 and performs an operation to change the progress status of the product replenishing for the corresponding product type to "completed." As a result, the control unit 51 of the mobile terminal 50 transmits a completion notification including the display area number corresponding to the product type together with his / her own identification information to the image processing device 10.

[0120] The control unit 11 of the image processing device 10 deletes the row linked to the received display area number from the shortage status management information. The control unit 11 generates new notification information from the updated shortage management information and display area management information, and transmits a shortage notification including the generated notification information to the mobile terminal 50 of each store clerk 3. As a result, the updated shortage notification screen 510 is displayed on each mobile terminal 50.

[0121] Furthermore, when the clerk 3 is unable to complete the replenishment because the replenishment target product is out of stock, he or she executes the above-mentioned application program on his or her own mobile terminal 50 and performs an operation to change the progress status of the replenishment of the product for the corresponding product type to "out of stock." This causes the control unit 51 of the mobile terminal 50 to send an out-of-stock notification including the display area number corresponding to the product type together with his or her own identification information to the image processing device 10.

[0122] The control unit 11 of the image processing device 10 updates the "progress" item of the row linked to the received display area number to "out of stock." The control unit 11 generates new notification information from the updated shortage management information and display area management information, and sends a shortage notification including the generated notification information to the mobile terminal 50 of each store clerk 3. As a result, the updated shortage notification screen 510 is displayed on each mobile terminal 50.

[0123] The above notification process allows the store clerk 3 to smoothly respond to the occurrence of a shortage of a product.

[0124] However, if the above-mentioned notification is made while serving a customer, it is possible that clerk 3 will not notice the notification and will not respond to the shortage for a long time. Also, clerk 3 may operate the shortage notification screen 510 in FIG. 12(b) to respond to the shortage, but may not be able to promptly replenish the product due to an emergency task that arises afterwards. In this case, too, the shortage will not be resolved for a long time.

[0125] To prevent such problems from occurring, in this embodiment, if the shortage state does not resolve within a predetermined reference time from the occurrence of the shortage state, a notification indicating that the shortage has not been resolved is sent from the control unit 11 of the image processing device 10 to the administrator's mobile terminal 50.

[0126] FIG. 13 is a flowchart showing a process of transmitting a shortage notification executed by the control unit 11 of the image processing device 10.

[0127] The control unit 11 refers to the shortage status management information in Fig. 11, compares each "shortage occurrence date and time" with the current time measured by its own clock function, and determines whether a product type area 202 (display area number) has occurred for which a reference time (e.g., 10 minutes) has elapsed since the "shortage occurrence date and time" (S301, S302). If such a product type area 202 (display area number) has occurred (S301: YES, S302: YES), the control unit 11 transmits a shortage unresolved notification to the mobile terminal 50 carried by the manager (S303). The shortage unresolved notification further includes information in which an excess flag has been added to identify the display area number for which the reference time has elapsed, in addition to the notification information contained in the shortage notification described above.

[0128] 14(a) and 14(b) are diagrams showing notification screens that are displayed on the mobile terminal 50 of the manager upon receiving a notification that the shortage has not been resolved.

[0129] As in the cases of FIGS. 12(a) and (b), a notification that the shortage has not been resolved is sent to the manager's mobile terminal 50 via a push notification. When the manager touches the icon of the application program in response to the push notification, notification items 521 and 522 shown in FIG. 14(a) are displayed on the display input unit 53. Notification item 521 includes a display showing the total number of product type areas 202 (product types) for which the shortage has not been resolved and the reference time has passed. Notification item 521 is the same as notification item 501 shown in FIG. 12(a).

[0130] When the administrator touches notification item 521, shortage unresolved notification screen 530 shown in FIG. 14(b) is displayed. On shortage unresolved notification screen 530, multiple display areas 531 separated by dashed lines are displayed in a manner that allows scrolling up and down. Each display area 531 displays the contents of the product type area 202 (product type) for which the shortage has not been resolved and the reference time has passed. Display area 531 includes a display of the same content as display area 511 in FIG. 12(b), and also includes an image indicating that the time has passed. If the "action status" in FIG. 11 is not unresolved, button 532 that displays the contents of "action status" is included, and if the "action status" is unresolved, button 532 is not included.

[0131] From the shortage unresolved notification screen 530, the manager can see which product types have not been addressed for a reference time, the response status, and whether or not there is someone to address the shortage. This allows the manager to urge the store clerk 3 to address the shortage of the product type. Furthermore, when the notification item 522 in FIG. 14(a) is touched, the shortage notification screen 510 in FIG. 12(b) is displayed. This allows the manager to see all product types currently experiencing shortages and their response status. The manager can take action to arrange for products for product types whose response status is "out of stock."

[0132] When the administrator refers to the display unit 13 of the image processing device 10, the screens of Figures 14(a) and (b) may be displayed on the display unit 13 rather than on the administrator's mobile terminal 50, or the screens of Figures 14(a) and (b) may be displayed on both the administrator's mobile terminal 50 and the display unit 13 of the image processing device 10.

[0133] <Effects of the embodiment> As shown in Figure 4, the image processing system 1 (image processing device 10) includes a setting unit 11a for setting a product display area R13 for each product type in an image P1 of a product display shelf 201, an extraction unit 11b for extracting a product area R11 corresponding to each product from the image P1, and a determination unit 11c for determining whether a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas R11 included in the display area R13 and the area of ​​the display area R13.

[0134] 6, the image processing method in the image processing device 10 sets a product display area R13 for each product type in the captured image P1 of the product display shelf 201, and in the process of Fig. 10, extracts a product area R11 corresponding to each product from the captured image P1 (S202), and determines whether or not the product is in short supply for each product type based on the correlation value between the total area of ​​the product areas R11 included in the display area R13 and the area of ​​the display area R13 (S203-S206). This image processing method is executed by the control unit 11 using a program stored in the storage unit 12.

[0135] As shown in FIG. 9, as products are removed from the product display shelf 201, the total area of ​​the product regions R11 included in the display area R13 decreases, and the correlation value between the total area of ​​the product regions R11 included in the display area R13 and the area of ​​the display area R13 changes. Therefore, this correlation value can accurately determine the product shortage status for the display area R13. Furthermore, unlike background images, the product images themselves do not differ from store to store. Therefore, the product regions R11 can be easily extracted from the captured image P1 using a general-purpose product recognition algorithm without the need for learning or tuning for each store. Therefore, the image processing system 1, image processing device 10, image processing method, and program according to this embodiment can easily and accurately detect the product shortage status.

[0136] Here, the correlation value can be the ratio of the total area to the area of ​​the display area R13, as in the above formula (1).

[0137] As described above, as products are removed from the product display shelf 201, the total area of ​​the product area R11 included in the display area R13 decreases, and the ratio of this total area to the area of ​​the display area R13 decreases. Therefore, by using this ratio as a correlation value, it is possible to accurately detect a product shortage.

[0138] As shown in Figures 5 and 6, the control unit 11 (setting unit 11a) extracts a product group area R12 in which products of the same product type are lined up consecutively from the product area R11 extracted from the captured image P1 (S104), and sets a display area R13 for the product type based on the extracted product group area R12 (S105).

[0139] According to this configuration, the control unit 11 (setting unit 11a) automatically extracts a product group area R12 for each product type from the captured image P1, so that the extracted product group area R12 can be set as is as the display area R13 for the corresponding product type, or the display area R13 for the corresponding product type can be easily set by accepting fine adjustments to the product group area R12 from the administrator (S106, S107).

[0140] As described above, the control unit 11 (setting unit 11a) extracts, from the product areas R11 extracted from the captured image P1, multiple product areas R11 that are identical or similar in at least one of area, height, width, height-to-width ratio, and shape and that are arranged consecutively as product group areas R12.

[0141] This configuration allows multiple product areas R11 corresponding to products of the same product type to be properly identified, thereby properly extracting the product group area R12 from the captured image P1 and properly setting the display area R13.

[0142] As shown in FIG. 4, the image processing system 1 (image processing device 10) includes a notification processing unit 11d that notifies the mobile terminal 50 of the user of the result of the shortage determination made by the determination unit 11c.

[0143] 12(a), (b) and 14(a), (b), this configuration makes it possible to notify users, such as store clerks, of product shortages, thereby enabling users to smoothly replenish the corresponding product display shelves 201 with the products that are running low.

[0144] <Change example 1> In the first modification, the control unit 11 (determining unit 11c) of the image processing device 10 sets the criteria for determining whether a product is in short supply.

[0145] The larger the size of the product, the fewer the number of products contained in the total product area. Therefore, when determining a product shortage by comparing the above correlation value with a fixed determination criterion, the number of products remaining for each product type when it is determined that a shortage has occurred may differ depending on the size of the products for each product type. In this case, for example, for a product type with individually small products, there will be more products remaining when it is determined that a shortage has occurred, and for a product type with individually large products, there will be fewer products remaining when it is determined that a shortage has occurred. If the number of remaining products is high, there is little need to replenish products, and if the number of remaining products is low, there is a need to replenish products urgently.

[0146] To prevent such occurrences as much as possible, in Modification Example 1, a criterion for determining whether each product type is in short supply is set based on the size of the product area R11 of each product type on the captured image P1. More specifically, the control unit 11 (determination unit 11c) sets the criterion for each product type based on the correlation between the size of the product area R11 of each product type on the captured image P1 and the size of the display area R13.

[0147] FIG. 15(a) is a flowchart showing the process of setting the criteria for determining a shortage state, which is executed by the control unit 11 (determining unit 11c).

[0148] 15(a) is preferably performed when products are displayed in an orderly manner throughout each product type area 202. This allows a product area R11 of standard size to be acquired for each product type from the captured image P1. The process for setting the determination criteria may be performed, for example, following the process for setting the display area R13 in FIG.

[0149] 15(a) is also performed for each camera 30. The control unit 11 extracts a product area R11 from the image P1 captured by the camera 30 (S401), and obtains the size of one product area R11 for each display area R13 (S402). The control unit 11 sets the determination criteria for each display area R13 based on the size of the product area R11 (S403).

[0150] The size of the product area R11 acquired in step S402 may be a representative size of all product areas R11 included in the display area R13. For example, the average value of the sizes of all product areas R11 included in the display area R13 may be acquired as the representative size, or the size of any one product area R11 may be acquired as the representative size.

[0151] The acquired size may be any parameter that indicates the size of the product region R11. For example, the acquired size may be the area of ​​the product region R11, or the width and height of the product region R11 may be acquired as the size of the product region R11.

[0152] The processing of step S403 may be, for example, the processing of Fig. 15(b). The control unit 11 acquires, for each display area R13, a parameter indicating the correlation between the size of the display area R13 and the size of the product area R11 acquired in step S402 (S412). This parameter may be, for example, the ratio of the size (e.g., area) of the product area R11 to the size (e.g., area) of the display area R13. The control unit 11 sets, for each display area R13, a determination criterion corresponding to the acquired parameter (S413).

[0153] In step S413, the control unit 11 refers to, for example, a table that associates parameters with judgment criteria. This table is stored in advance in the storage unit 12. Each judgment criterion in the table (such as the threshold value described above) is set so that the number of remaining products in the product type area 202 when it is determined that a shortage has occurred based on the corresponding parameter is as close as possible between the judgment criteria. In this case, the remaining number may be several (for example, 3 to 6). The table may be included in the application program described above. Alternatively, a user such as an administrator may be able to generate the table via a setting screen displayed by executing the application program.

[0154] In step S413, the control unit 11 acquires, from the table, the determination criteria corresponding to the parameters acquired in step S412, and sets the acquired determination criteria as the determination criteria to be used for the corresponding display area R13. In this way, the determination criteria are set for each display area R13. The control unit 11 executes the processes of steps S205 and S206 in FIG. 10 using the determination criteria thus set for each display area R13.

[0155] <Effects of Change Example 1> As shown in FIGS. 15(a) and 15(b), the control unit 11 (determination unit 11c) sets a criterion for determining whether a product type is in short supply, based on the size of the product area R11 for each product type in the captured image P1. This allows the number of remaining products for each product type to be approximately the same when determining whether a product type is in short supply. Therefore, for each product type, the store clerk 3 is notified of the shortage when the remaining number reaches approximately the same number. This allows the store clerk 3 to replenish the products appropriately and smoothly.

[0156] The proportion of one product area R11 in the display area R13 varies depending on the size of the display area R13. Therefore, the larger the display area R13, the greater the number of remaining products for the same correlation value. Therefore, if the judgment criteria were set based solely on the size of the product area R11, the number of remaining products when it is determined that a shortage has occurred may vary depending on the size of the display area R13. In contrast, in the process of FIG. 15(a), the judgment criteria are set based on the correlation between the size of the product area R11 and the size of the display area R13. Therefore, the number of remaining products when it is determined that a shortage has occurred can be made similar between product types, and users such as store clerks can be notified of the shortage at an appropriate time.

[0157] If the size of the display area R13 does not vary significantly between product types, the judgment criteria for each product type may be set based only on the size of the product area R11. In this case, too, the number of remaining products when it is determined that a shortage has occurred can be made similar between product types.

[0158] As described above, the area of ​​the product region R11 can be used as the size of the product region R11 used to set the determination criteria.

[0159] According to this configuration, the determination criteria can be smoothly set based on the area of ​​the product region R11, which reflects the size of the entire product region R11.

[0160] <Change example 2> In the above embodiment, as shown in FIG. 13, if the shortage state is not resolved within a reference time, a shortage resolution notification is sent to the manager's mobile terminal 50 to notify the manager of the shortage state. However, if a shortage state occurs for a specific product type such as a sale item, a best-selling item, or a high-priced item, it is preferable that the shortage notification be sent not only to the mobile terminal 50 of the store clerk 3 but also to the mobile terminal 50 of the manager, so that the product is replenished more reliably.

[0161] In this case, the flowchart in Fig. 10 is modified as shown in Fig. 16. For convenience, Fig. 16 shows only the processing steps from step S208 onwards. In Modification Example 2, steps S210 and S211 are added. The other processing steps are the same as those in the flowchart in Fig. 10.

[0162] If the newly shortage-prone product type is a specific product type (S210: YES), the control unit 11 (notification processing unit 11d) further transmits a shortage notification to the manager's mobile terminal 50 (S211). As a result, a screen similar to that shown in FIGS. 12(a) and 12(b) is displayed on the manager's mobile terminal 50. This allows the manager to quickly grasp that a shortage has occurred for the specific product type, and to smoothly prompt the store clerk 3 to take action such as replenishing the product.

[0163] Here, the specific product type may be further linked to the display area management information of Fig. 8. For example, the input screen displayed in step S106 of Fig. 6, which includes the display area setting image P2 (Fig. 7) and the product information input fields, may further include an input field for designating each product type as a specific product type. The administrator may input a designation for a specific product type, such as a sale item, a best-selling item, or a high-priced item, on this input screen. This causes the control unit 11 to attach a specific product type flag to the corresponding product type in the display area management information of Fig. 8. The control unit 11 may make the determination of step S210 of Fig. 16 based on this flag.

[0164] The designation of the specific product type does not necessarily have to be performed at the timing when the flowchart of Fig. 6 is executed, but may be performed at any timing by the manager. Also, a predetermined store clerk 3 may designate the specific product type together with the manager.

[0165] <Other change examples> In the above embodiment, the shortage status determination process shown in Fig. 10 is executed by the image processing device 10, but this determination process may also be performed by the edge PC 20. In this case, the display area management information registered in the image processing device 10 may be transmitted to and stored in the corresponding edge PC 20, and the shortage status management information in Fig. 11 may be managed by the edge PC 20. Alternatively, a server connected to an external network may manage this information, and the server may perform the process shown in Fig. 10. In this case, the server may notify the store clerk 3 and the manager's mobile terminal 50 of the shortage status via the external network.

[0166] In the above embodiment, formulas (1) to (3) are used as examples of formulas for calculating the correlation value between the total area of ​​the product areas R11 included in the display area R13 and the area of ​​the display area R13, but the method for calculating the correlation value is not limited to these. For example, the correlation value may be calculated by multiplying the numerator or denominator on the right side of formula (1) by a predetermined coefficient. The correlation value may be any value that indicates the correlation between the total area of ​​the product areas R11 included in the display area R13 and the area of ​​the display area R13.

[0167] Furthermore, in the above embodiment, a rectangular area including an image of a product is extracted as each product area, but the method of extracting a product area is not limited to this. For example, the outline of the product may be extracted from the image of the product, and the area of ​​this outline may be extracted as the product area. Alternatively, an area obtained by excluding a peripheral portion of a predetermined width from the outline area may be extracted as the product area. The extracted product area may be an area having a size corresponding to the size of each product in the captured image.

[0168] Furthermore, the configuration of the product shelves 2 and the layout of the store S1 shown in the above embodiment are merely examples, and other configurations and layouts may be used. Similarly, the various screens are not limited to the configurations shown in Figures 12(a), (b) and 4(a), (b), and may be screens with other configurations.

[0169] In addition, the embodiments of the present invention can be modified as appropriate within the scope of the claims. [Explanation of symbols]

[0170] 1. Image processing system 10 Image processing device 11 Control section 11a Setting section 11b Extraction part 11c Judgment section 11d Notification processing section 50 Mobile Devices P1 Captured image P2 Display area setting image P11 Shelf image P12 Product image R11 Product area R12 Product group area R13 Display area

Claims

1. a setting unit for setting a product display area for each product type in a captured image of the product display shelf; an extraction unit that extracts a product area corresponding to each product from the captured image; a determination unit that determines whether or not a product is in short supply for each product type based on a correlation value between a total area of ​​the product areas included in the display area and an area of ​​the display area, An image processing system comprising:

2. 2. The image processing system according to claim 1, the correlation value is a ratio of the total area to the area of ​​the display area; An image processing system comprising:

3. 2. The image processing system according to claim 1, the determination unit sets a criterion for determining the shortage state for each of the product types based on the size of the product area for each of the product types on the captured image. An image processing system comprising:

4. 4. The image processing system according to claim 3, the determination unit sets the determination criterion for each of the product types based on a correlation between a size of the product area of ​​each of the product types and a size of the display area on the captured image. An image processing system comprising:

5. 4. The image processing system according to claim 3, The size of the product region used to set the judgment criteria is the area of ​​the product region. An image processing system comprising:

6. 2. The image processing system according to claim 1, the setting unit extracts a product group area in which products of the same product type are lined up consecutively from the product area extracted from the captured image, and sets the display area for the product type based on the extracted product group area. An image processing system comprising:

7. 7. The image processing system according to claim 6, the setting unit extracts, from the product regions extracted from the captured image, a plurality of product regions that are identical or similar in at least one of area, height, width, height / width ratio, and shape and that are arranged consecutively, as the product group region; An image processing system comprising:

8. 8. The image processing system according to claim 1, a notification processing unit that notifies a user's mobile terminal of the result of the shortage state determination by the determination unit; An image processing system comprising:

9. a setting unit for setting a product display area for each product type in a captured image of the product display shelf; an extraction unit that extracts a product area corresponding to each product from the captured image; a determination unit that determines whether or not a product is in short supply for each product type based on a correlation value between a total area of ​​the product areas included in the display area and an area of ​​the display area, 1. An image processing device comprising:

10. a step of setting a product display area for each product type in a captured image of the product display shelf; extracting product areas corresponding to each product from the captured image; determining whether or not a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area, An image processing method comprising:

11. a function for setting product display areas for each product type in a captured image of a product display shelf; a function of extracting a product area corresponding to each product from the captured image; A program that causes a computer to execute a function of determining whether or not a product is in short supply for each product type based on a correlation value between the total area of ​​the product areas included in the display area and the area of ​​the display area.

Images