Information processing device, product monitoring system, and program

Abstract

To provide an information processing device, a product monitoring system, and a program capable of smoothly notifying a store manager such as a clerk when an abnormality may have occurred in a product returned to a product shelf by a customer.SOLUTION: A product monitoring system 1 comprises an act identification unit 11a which identifies, on the basis of a captured image of an imaging unit 20 capable of imaging a product on a product shelf and a customer accessing the product, an act in which the customer takes the product out of the product shelf and then returns the product to the product shelf; a deterioration determination unit 11b which determines, on the basis of the captured image of the imaging unit 20, whether or not the act satisfies a predetermined estimation condition for estimating a possibility that deterioration of the product has occurred; and a deterioration notification unit 11c which executes a notification process for notifying a store manager of a target product of the act on the basis of the estimation condition being satisfied.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a product monitoring system, and a program for monitoring products displayed on a product shelf. [Background technology]

[0002] In stores, products displayed on shelves can deteriorate due to customer purchasing behavior. In such cases, store staff must promptly check the condition of the product and collect it as appropriate. For example, if a frozen product is removed from a freezer shelf and placed on a room-temperature shelf, the store staff must promptly check the condition of the product and return it to the freezer shelf.

[0003] The following Patent Document 1 describes a shelf monitoring system that uses a temperature sensor to monitor whether a frozen or refrigerated product has been left on a room-temperature shelf. In this system, a store server analyzes temperature distribution data acquired by the temperature sensor. If the store server determines through this analysis that a frozen or refrigerated product has been left on a room-temperature shelf, it transmits leaving information indicating this to a notification receiving device. This notifies a store clerk that the product has been left on a room-temperature shelf. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-176265 Summary of the Invention [Problem to be solved by the invention]

[0005] However, a customer may remove a frozen product from a freezer shelf and place it in a shopping cart, but then, after a certain amount of time has passed, return the frozen product to the original freezer shelf. A customer may also accidentally drop a product while removing it from a room-temperature shelf, and then return the product to the original shelf. In such cases, it is preferable for the store clerk to be able to quickly check whether the returned product has deteriorated or is abnormal. The system of Patent Document 1 cannot notify the store clerk of such a case, and the store clerk cannot quickly check the condition of the product.

[0006] In view of such problems, the present invention aims to provide an information processing device, a product monitoring system, and a program that can smoothly notify store managers such as store clerks when an abnormality may have occurred in a product that has been returned to the shelf by a customer. [Means for solving the problem]

[0007] A first aspect of the present invention relates to an information processing device. The information processing device according to this aspect includes: an action identification unit that identifies, based on an image captured by an imaging unit capable of capturing an image of a product on a product shelf and a customer accessing the product, an action of a customer removing a product from a product shelf and then returning the product to the product shelf; a deterioration determination unit that determines, based on the image captured by the imaging unit, whether the action satisfies a predetermined inference condition for inferring a possibility that the product has deteriorated; and a deterioration notification unit that, based on the inference condition being satisfied, executes a notification process to notify a store manager of information related to the product that was the target of the action.

[0008] According to the information processing device of this aspect, the action of a customer removing and then returning a product can be identified based on captured images of the product on the shelf and the customer accessing the product. This allows this action to be identified for products displayed on any shelf, regardless of whether the product has a specific temperature characteristic, such as refrigerated or frozen products. Furthermore, whether the action satisfies predetermined inference conditions based on the captured images determines whether the target product may have deteriorated. This determination can be made for products displayed on any shelf. When the inference conditions are satisfied, information about the target product that may have deteriorated is reported to a store manager, allowing store staff or other store managers to smoothly notify them of the possibility that an abnormality has occurred in a product returned to the shelf by a customer. This allows the manager to smoothly and quickly check the condition of the target product.

[0009] In the information processing device of this aspect, the estimation conditions may include that the target product is a refrigerated product or a frozen product, and that the time between the customer removing the target product from the shelf and returning it to the shelf exceeds a threshold time.

[0010] According to this configuration, if the target product is a refrigerated or frozen product, and the time from when it was taken out to when it was returned exceeds a threshold time, and there is a possibility that the target product has deteriorated, information about the target product is notified to a manager such as a store clerk. This allows the manager to smoothly check the condition of the target product, which may have deteriorated, such as melting, due to being exposed to a room temperature environment for a long period of time.

[0011] In the information processing device of this aspect, the deterioration determination unit determines whether a reason for deterioration has occurred in the target product based on the captured image, and the estimation condition may include that a reason for deterioration has occurred in the target product.

[0012] According to this configuration, information about products that may have been damaged or broken due to a deterioration cause such as a drop is notified to a manager such as a store clerk, allowing the manager to smoothly check whether the target products have been damaged or broken due to a deterioration cause.

[0013] In the information processing device of this aspect, the deterioration determination unit determines whether or not an abnormality has occurred in the appearance of the target product before it is removed and after it is returned based on the captured image, and the inferred condition may include that an abnormality has occurred in the appearance.

[0014] According to this configuration, if an abnormality occurs in the appearance of the target product before it is taken out and after it is returned, information about the target product is notified to a manager such as a store clerk, thereby enabling the manager to smoothly check whether the target product has deteriorated.

[0015] In the information processing device of this aspect, the action identification unit may be configured to obtain the position to which the target product was returned from the captured image, and the deterioration notification unit may be configured to execute a process to display a notification screen including information indicating the position as information regarding the target product.

[0016] With this configuration, a manager such as a store clerk can refer to the notification screen to ascertain the location where the possibly deteriorated product has been returned, allowing the manager to smoothly and quickly check the condition of the product.

[0017] In this configuration, the behavior identification unit can be configured to obtain a facial image of the customer who performed the behavior from the captured image, and the deterioration notification unit can be configured to display the facial image on the notification screen.

[0018] With this configuration, a manager such as a store clerk can recognize the face of the customer who performed the behavior that satisfies the estimation condition from the notification image. Therefore, after checking the condition of the target product, the manager can ask the customer who performed the behavior to take caution.

[0019] In this case, the deterioration notification unit may be configured to further execute a process of displaying a facial image of a customer for whom the number of times of the behavior that satisfies the estimation condition exceeds a predetermined threshold.

[0020] This configuration allows a manager, such as a store clerk, to identify customers who repeatedly perform actions that satisfy the inference conditions, and can take measures such as strengthening monitoring of such customers when they visit the store.

[0021] A second aspect of the present invention relates to a merchandise surveillance system, which comprises the information processing device according to the first aspect and the imaging unit.

[0022] A third aspect of the present invention relates to a program that causes a control unit of an information processing device that can communicate with an imaging unit that can capture images of products on a shelf and customers accessing the products to execute the functions of the behavior recognition unit, deterioration determination unit, and deterioration notification unit of the first aspect.

[0023] According to the second and third aspects of the present invention, the same effects as those of the first aspect can be achieved. [Effects of the Invention]

[0024] As described above, according to the present invention, it is possible to provide an information processing device, a product monitoring system, and a program that can smoothly notify store managers such as store clerks if an abnormality occurs in a product that has been returned to the shelf by a customer.

[0025] The effects and significance of the present invention will become more apparent from the following description of the embodiments, however, the embodiments shown below are merely examples of how the present invention can be implemented, and the present invention is not limited to the embodiments described below. [Brief explanation of the drawings]

[0026] [Figure 1]FIG. 1 is a diagram showing the configuration of a merchandise surveillance system according to an embodiment. [Figure 2] FIG. 2 is a plan view schematically illustrating an example of a store layout according to the embodiment. [Figure 3] FIG. 3 is a front view schematically showing the configuration of a product shelf according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of an information processing device according to the embodiment. [Figure 5] 5(a) and 5(b) are diagrams showing the configurations of product management information and shelf management information stored in a storage unit of an information processing device according to an embodiment. [Figure 6] FIG. 6 is a diagram showing the configuration of deterioration management information according to the embodiment. [Figure 7] FIG. 7 is a flow chart illustrating a process for merchandise surveillance according to an embodiment. [Figure 8] FIG. 8 is a diagram showing the configuration of a notification screen for notifying that a product may have deteriorated, according to the embodiment. [Figure 9] FIG. 9 is a diagram showing the configuration of customer management information according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing the customer management information registration process according to the embodiment. [Figure 11] FIG. 11 is a flowchart illustrating a process for notifying a customer whose number of actions satisfying an estimation condition has exceeded a predetermined threshold, according to the embodiment. [Figure 12] FIG. 12 is a diagram showing the configuration of a notification screen for notifying a customer whose number of actions satisfying an estimation condition has exceeded a predetermined threshold, according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing a process for monitoring merchandise according to the first modification. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0028] FIG. 1 is a diagram showing the configuration of a merchandise surveillance system 1 according to an embodiment.

[0029] The product monitoring system 1 includes an information processing device 10, an imaging unit 20, and a transmission / reception device 30. The imaging unit 20 and the transmission / reception device 30 are installed in a store where product shelves 2 are installed. The information processing device 10 is capable of communicating with the imaging unit 20 and the transmission / reception device 30 via a network 40. The information processing device 10 is installed in the same store where the product shelves 2 are installed. The network 40 is a LAN (Local Area Network) installed in the store. The information processing device 10 may also be installed in a facility outside the store. In this case, the network 40 is an external communications network such as the Internet.

[0030] The information processing device 10 manages information relating to the store and the product shelves 2. As part of this management, the information processing device 10 monitors the possibility of deterioration of products on the product shelves 2. If this monitoring determines that there is a possibility that a product has deteriorated, the information processing device 10 executes a notification process to prompt the user to check the product.

[0031] The information processing device 10 is, for example, a server computer or a personal computer. The information processing device 10 may also be a dedicated device. The information processing device 10 may include interfaces such as a display, a keyboard, a mouse, and a speaker (not shown in FIG. 1).

[0032] The imaging unit 20 is arranged so as to be able to capture images of the products displayed on the product shelf 2 and the customers 3 accessing the products. Specifically, the imaging unit 20 is a camera. Multiple imaging units 20 may be arranged to capture images of the products and the customers 3. For example, an imaging unit 20 may be arranged to capture an image of the entire front of the product shelf 2, and an imaging unit 20 may be arranged to capture an image of the customers 3 facing the product shelf 2 from the front. The imaging unit 20 transmits the captured images to the information processing device 10. By analyzing the captured images, it is possible to detect that a product has been taken out of each shelf of the product shelf 2 by a customer 3, and that the taken-out product has been returned to the product shelf 2.

[0033] The transmitting / receiving device 30 is a communication device for wirelessly communicating with the mobile terminal 50. The transmitting / receiving device 30 is installed on the ceiling of the store, the top surface of the product shelf 2, or the like. The transmitting / receiving device 30 may be a wireless LAN router. The mobile terminal 50 is, for example, a mobile phone such as a smartphone. The mobile terminal 50 is distributed to each store clerk 4 and carried by the corresponding store clerk 4.

[0034] FIG. 2 is a plan view schematically showing an example of the layout of the store S1.

[0035] Here, the store S1 is divided into a sales floor S11 and a checkout area S12. A plurality of product shelves 2 are installed in the sales floor S11. A plurality of imaging units 20 (cameras) are installed facing each of the product shelves 2. The sales floor S11 has an entrance S13. The checkout area S12 is separated from the sales floor S11 by a counter 5. Two cash registers 6 are installed at the counter 5. An imaging unit 20 is also installed that images a customer 3 making a checkout at the cash register 6, and another imaging unit 20 that images a customer 3 entering the store from the entrance S13.

[0036] In the example of FIG. 2, two store clerks 4 are working at store S1. Here, the two store clerks 4 are each in charge of a cash register 6. In addition, one customer 3 is visiting the store. An information processing device 10 is installed at the back of the checkout area S12. The store clerk 4 can view the display screen of the information processing device 10 at any time in the checkout area S12.

[0037] As will be described later, a notification screen for notifying that a product may have deteriorated is displayed on the information processing device 10. By looking at this notification screen, the store clerk 4 can identify the product that may have deteriorated, and can go to the product shelf 2 where the product is displayed to check the condition of the product. Such a notification screen may also be displayed on the mobile terminal 50 of the store clerk 4.

[0038] FIG. 3 is a front view schematically showing the configuration of the product shelf 2. As shown in FIG.

[0039] The product shelf 2 includes a plurality of shelves 201 for displaying products. Each shelf 201 is divided into a plurality of display areas 202. A label 203 indicating the position of each shelf 201 on the product shelf 2 is attached to the right side of each shelf 201. Also, a label 204 indicating the position of each display area 202 on the shelf 201 is attached to the underside of each display area 202. For example, a number or code indicating the position is displayed on the labels 203 and 204.

[0040] Furthermore, to the right of each label 204, a tag 205 is provided to display the name and price of the product displayed in each display area 202. Also, a label 206 indicating the identification code of the product shelf 2 is attached to the upper left corner of the product shelf 2. The identification code of the product shelf 2 is, for example, a number assigned consecutively to all the product shelves 2 in the store S1.

[0041] FIG. 4 is a block diagram showing the configuration of the information processing device 10. As shown in FIG.

[0042] 4 also illustrates the imaging unit 20, the transmitting / receiving device 30, the network 40, and the mobile terminal 50. Although two mobile terminals 50 are shown here, in reality there are as many mobile terminals 50 as there are store clerks 4 working in the store S1.

[0043] The information processing device 10 includes a control unit 11, a storage unit 12, an interface 13, and a communication unit 14.

[0044] The control unit 11 includes an arithmetic processing circuit such as a CPU (Central Processing Unit) and controls each unit according to a program stored in the storage unit 12. The storage unit 12 includes memories such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk, and stores the program. The storage unit 12 is also used as a work area when the control unit 11 executes the program.

[0045] The interface 13 is an input / output interface that presents information to the store clerk 4 and receives input from the store clerk 4 under the control of the control unit 11. The interface 13 includes a display such as a liquid crystal panel, input means such as a keyboard and a mouse, and a speaker that outputs predetermined sounds. The communication unit 14 communicates with the imaging unit 20 and the mobile terminal 50 under the control of the control unit 11. As described above, communication with the mobile terminal 50 is performed via the transmission / reception device 30.

[0046] In this embodiment, a program for product monitoring is stored in the storage unit 12. This program causes the control unit 11 to execute the functions of an action recognition unit 11a, a deterioration determination unit 11b, and a deterioration notification unit 11c. This program includes an image analysis engine for analyzing captured images to realize these functions.

[0047] The action identification unit 11a identifies the action of the customer 3 taking a product from the product shelf 2 and then returning the product to the product shelf 2, based on images captured by the imaging unit 20, which is capable of capturing images of the product on the product shelf 2 and the customer 3 accessing the product. Here, the product shelf 2 to which the customer 3 returns the product does not necessarily have to be the same product shelf 2 from which the customer 3 took the product. The action identification unit 11a analyzes the captured images input from each of the multiple imaging units 20 shown in Figure 2, and identifies whether the customer 3 has returned the product to the display area 202 of any of the product shelves 2.

[0048] The deterioration determination unit 11b tracks the actions of a customer 3 from taking a product from the shelf 2 to returning the product to the shelf 2 based on the images captured by the corresponding imaging unit 20, and determines whether or not a predetermined estimation condition for estimating the possibility that the product has deteriorated is met during this action.

[0049] Here, the inference conditions may include, for example, that the target product that was the subject of the above-mentioned act is a refrigerated product or a frozen product, and that the time from when the customer 3 took the target product off the shelf 2 until when they returned it to the shelf 2 exceeds a threshold time. If these conditions are met, it is inferred that there is a possibility that the target product has deteriorated.

[0050] The inference condition may also include a determination based on the captured image that an abnormality has occurred in the appearance of the target product before it was taken out and after it was returned. If this condition is met, it is also inferred that the target product may have deteriorated. This inference condition applies to all types of products, regardless of whether the product is frozen or refrigerated.

[0051] However, the estimation conditions are not limited to these, and other conditions may be included as long as it is possible to estimate that the target product may have deteriorated based on the captured image.

[0052] Based on the satisfaction of the above-mentioned estimation conditions, the deterioration notification unit 11c executes a notification process for notifying a store manager, such as a store clerk 4, of the target product that was the subject of the above-mentioned action. As described above, in this embodiment, as the notification process, a notification screen for notifying that there is a possibility that deterioration has occurred in the product is displayed from the interface 13 of the information processing device 10.

[0053] As described above, the deterioration notification unit 11c may further cause the mobile terminal 50 of the store clerk 4 to display a similar notification screen via the communication unit 14. In this case, an application program for communicating with the information processing device 10 and displaying this notification screen is installed in the mobile terminal 50.

[0054] 5(a) and 5(b) are diagrams showing the configurations of the product management information and shelf management information stored in the storage unit 12 of the information processing device 10. FIG.

[0055] Referring to FIG. 5(a), the product management information is configured by linking a product code, which is product identification information, a product name, and a product type and category.

[0056] With reference to FIG. 5(b), shelf management information is stored in the storage unit 12 for each product shelf 2. That is, the shelf management information of FIG. 5(b) is stored in the storage unit 12 for each product shelf 2 installed in the store S1. The shelf management information is configured by linking together a shelf code, which is identification information of the shelf 201 on the product shelf 2, the name of that shelf 201 (product shelf name), a position code indicating the position of the display area 202 on that shelf 201, and the product codes of the products displayed in that display area 202. The shelf code corresponds to the symbol or number shown on the label 203 in FIG. 3, and the position code corresponds to the symbol or number shown on the label 204 in FIG. 3.

[0057] The product management information in Fig. 5(a) and the shelf management information in Fig. 5(b) are registered in the information processing device 10 by the manager of the store S1 as needed, depending on the purchase of products and the layout settings of the product shelves. In addition, the storage unit 12 may store information for communicating with the mobile terminals 50 carried by the respective store clerks 4.

[0058] FIG. 6 is a diagram showing the structure of deterioration management information for managing the possibility of deterioration of a product taken out from the product shelf 2. As shown in FIG.

[0059] The deterioration management information is stored in the storage unit 12 and updated as needed. The deterioration management information is configured by linking together the following items: product code, product name, variety, item, removal time, return time, return position, reason for deterioration, appearance abnormality, notification, and facial image. The product code, product name, variety, and item are the same as the items in the product management information in FIG. 5(a).

[0060] "Removal time" is the time when the corresponding product is removed from the product shelf 2, and "return time" is the time when the corresponding product is returned to the product shelf 2. "Return position" is information indicating the position on the product shelf 2 to which the product is returned, and is specified by the number or code displayed on the labels 203, 204, and 206 in Figure 3.

[0061] A "reason for deterioration" is a reason for which it can be determined that a product may have deteriorated based on the captured image. For example, a "reason for deterioration" may be when the time between when a frozen or refrigerated product is taken out of the product shelf 2 and when it is returned exceeds a predetermined threshold time. Alternatively, a "reason for deterioration" may be when the product is dropped or opened between when it is taken out and when it is returned. However, the reason for deterioration is not limited to this, and may include other reasons such as the product being soiled.

[0062] "Appearance abnormality" is an item that indicates whether an abnormality has occurred in the appearance of the product before it was taken out and after it was returned. "Notification" is flag information that indicates whether or not to execute notification processing to notify the store clerk that the corresponding product may have deteriorated. "Facial image" is an image that has been cut out from the captured image to show the facial area of the customer 3 who took out and returned the corresponding product. The "facial image" may also be stored in association with the characteristic values of parameters that indicate the characteristics of the facial image.

[0063] When the control unit 11 detects, by the function of the action recognition unit 11a, based on the captured image from the imaging unit 20, that a new product has been removed from the product shelf 2, it adds a new row to the deterioration management information to store information about that product. At this time, the control unit 11 extracts a facial image of the customer 3 who removed the product from the product shelf 2 from the captured image and registers it in the "Facial Image" field of the new row.

[0064] Furthermore, the control unit 11, using the function of the action recognition unit 11a, identifies, from the captured image, the product shelf 2 from which the product was taken and the display area 202 of the product on that product shelf 2. This identification is performed by analyzing the captured image of the corresponding product shelf 2 and reading the information on the labels 203, 204, and 206 in Fig. 3.

[0065] The control unit 11 extracts the location code corresponding to the identified location of the product shelf 2 and display area 202 from the shelf management information in Fig. 5(b), and identifies the product code corresponding to the extracted location code in the shelf management information.The control unit 11 then extracts the product name, variety, and item corresponding to the identified product code from the product management information in Fig. 5(a), and registers this extracted information in each item of a new row in the deterioration management information.

[0066] Furthermore, the control unit 11, using the function of the deterioration determination unit 11b, obtains the time when the product was removed from its own clock circuit and registers the obtained time in the "Removal Time" field of the new row. In this way, the registration of product management information at the time of product removal is completed. Thereafter, the control unit 11, using the function of the deterioration determination unit 11b, tracks the behavior of the customer 3 who removed this product. The tracking is performed using information that can identify the customer 3 that can be obtained from the captured image, such as an image of the customer 3's face, as well as the color of their clothing.

[0067] In this tracking, the control unit 11 uses the function of the deterioration determination unit 11b to determine whether or not a deterioration event such as dropping or opening of the product has occurred based on the image captured by the corresponding imaging unit 20. The control unit 11 registers the deterioration event that occurred in this tracking in the "Reason for Deterioration" field of a new row.

[0068] Furthermore, when the control unit 11 identifies from the captured image through this tracking that the product has been returned to the product shelf 2, it uses the function of the deterioration determination unit 11b to obtain from its own clock circuit the time at which the product was returned, and registers the obtained time in the "return time" field of the new row. Furthermore, just as when a product is removed, the control unit 11 uses the function of the action identification unit 11a to identify the product shelf 2 to which the product has been returned and its position in the display area 202, and registers information related to the identified product shelf 2 and position (shelf number, position code, etc.) in the "return position" field of the new row.

[0069] Furthermore, the control unit 11, using the function of the deterioration determination unit 11b, compares an image of the product when it is removed from the product shelf 2 with an image of the product when it is returned to the product shelf 2, and determines whether any abnormality has occurred in the appearance of the product between removal and return.

[0070] Here, the image of a product when it is taken out from the shelf 2 may be, for example, an image of the product in the state in which it was displayed on the shelf 2 immediately before it was taken out. In this case, among the images of each product in the state in which it was displayed immediately before it was taken out, an image of the product that was in the position that was missing after it was taken out is stored in the storage unit 12 as the image of the product when it was taken out from the shelf 2. Also, the image of a product that has been returned to the shelf 2 may be, for example, an image of the product in the state in which it was displayed on the shelf 2 immediately after it was returned. In this case, an image of the product that was added immediately after it was returned to the images of each product in the state in which it was displayed immediately before it was returned is acquired as the image of the product returned to the shelf 2.

[0071] In addition, an image obtained by extracting the area of the product being held by the customer 3 from the captured image during the act of taking out may be stored in the storage unit 12 as an image of the product being taken out from the product shelf 2. Also, an image obtained by extracting the area of the product being held by the customer 3 from the captured image during the act of putting back may be acquired as an image of the product being returned to the product shelf 2.

[0072] When the control unit 11 determines that an abnormality has occurred in the appearance of the product by comparing the images of the product before and after removal, it registers flag information indicating this in the "appearance abnormality" field of a new row.

[0073] Furthermore, when the control unit 11 determines, through the function of the deterioration determination unit 11b, that the actions of the customer 3 from the time the product is taken out to the time the product is returned satisfy predetermined inference conditions for inferring the possibility that the product has deteriorated, the control unit 11 uses the function of the deterioration notification unit 11c to register flag information indicating that notification is necessary in the "Notification" field of the new row. The inference conditions for this determination are as described above. This completes the registration process for the new row.

[0074] In Figure 6, in the first line from the top, the time limit for the frozen food product has been exceeded, so the estimation condition is met and flag information indicating notification is required is registered in the "Notification" field. Also, in the fifth line from the top, an abnormality in the product's appearance has been detected, so flag information indicating notification is required is registered in the "Notification" field. In the third line from the top, although a deterioration event such as a drop was detected, no abnormality in appearance was detected, so flag information indicating notification is not registered in the "Notification" field.

[0075] Thereafter, when payment is made for any product registered in the deterioration management information, the one-line information for that product is deleted from the deterioration management information. Whether or not payment has been made for that product is determined, for example, based on an image captured by the imaging unit 20 that captures an image of the customer 3 making payment at the cash register 60 in FIG. 2. The control unit 11 of the information processing device 10 extracts a facial image of the customer 3 included in the image captured by the imaging unit 20. When the control unit 11 determines from a subsequent captured image that the customer 3 has completed payment, it extracts facial images in the deterioration management information that match the facial image extracted at the time of payment, and deletes lines containing all of the extracted facial images from the deterioration management information.

[0076] For rows in which flag information indicating notification is registered in the "Notification" field, the information is registered in the customer management information before being deleted upon settlement, as will be described later. In addition, when the information processing device 10 is shut down, all of the deterioration management information remaining in the storage unit 12 may be deleted at once.

[0077] FIG. 7 is a flowchart showing a process for product monitoring executed by the control unit 11 of the information processing device 10.

[0078] In the flowchart of Fig. 7, the processes of steps S101, S102, and S106 are executed by the function of the action identification unit 11a of Fig. 4, the processes of steps S103 to S106 and S108 to S111 are executed by the function of the deterioration determination unit 11b of Fig. 4, and the processes of steps S112 and S113 are executed by the function of the deterioration notification unit 11c of Fig. 4. In the following, these functions will not be particularly mentioned, and the description will be made assuming that the control unit 11 executes the processes of Fig. 7 using these functions.

[0079] The control unit 11 acquires and analyzes captured images from each of the image capturing units 20 in Fig. 2 at a predetermined cycle (S101), and determines whether or not the customer 3 has removed an item from any of the product shelves 2 (S102). When the customer 3 removes an item from a product shelf 2 (S102: YES), the control unit 11 extracts a facial image of the customer 3 from the image captured by the corresponding image capturing unit 20, and registers the extracted facial image and the time of removal in a new row of the deterioration management information (S103). At this time, the control unit 11 acquires information identifying the removed item from the image captured by the corresponding image capturing unit 20, and registers this information in the new row of the deterioration management information. In the "Notification" field of the new row, flag information indicating "unnecessary" is registered by default.

[0080] Thereafter, the control unit 11 tracks the behavior of the customer 3 (S104), and appropriately registers the reason for deterioration in the new row of the deterioration management information (S105).

[0081] The control unit 11 repeatedly executes the processes of steps S104 and S105 while the removed product is not returned to any of the product shelves 2 (S106: NO) and payment for the product is not made (S107: NO). Thereafter, when payment for the removed product is made (S107: YES) without the removed product being returned to any of the product shelves 2 (S106: NO), the control unit 11 ends the process of Fig. 7. At this time, the control unit 11 deletes the row corresponding to the product from the deterioration management information, as described above.

[0082] When the removed product is returned to any of the product shelves 2 (S106: YES), the control unit 11 registers the return time and return location in the new row of the deterioration management information (S108). Then, as described above, the control unit 11 compares the appearance of the product at the time of removal with the appearance of the returned product (S109), and if there is an appearance abnormality, registers flag information for that in the new row of the deterioration management information (S110). If there is no appearance abnormality, the control unit 11 skips step S110.

[0083] The control unit 11 determines whether the above-mentioned estimation conditions are satisfied (S111). If the estimation conditions are not satisfied (S111: NO), the control unit 11 ends the processing of FIG. 7. On the other hand, if the estimation conditions are satisfied (S111: YES), the control unit 11 registers flag information indicating that notification is necessary in the new row of the deterioration management information (S112). Then, the control unit 11 executes notification processing to notify the store clerk 4 that the returned product may have deteriorated (S113). In these cases, the row corresponding to the product is deleted from the deterioration management information upon payment for the product, as described above.

[0084] FIG. 8 is a diagram showing the configuration of the notification screen 100 displayed on the interface 13 of FIG. 4 in the notification process of step S113 of FIG.

[0085] The notification screen 100 includes a message 101 informing the customer that the product returned to the product shelf 2 may have deteriorated and urging the customer to check the product, a display item 102 displaying the product name, the product shelf to which the product was returned and the location to which it was returned, a display item 103 displaying the reason for the deterioration of the product, an image 104 displaying the product shelf 2 to which the product was returned together with the product's display area 202, an image 105 of the face of the customer 3 who performed the action on the product, and an OK button 106.

[0086] In the product shelf column of the display item 102, for example, an identification symbol attached to that product shelf 2 is displayed. In the image 104, the display area 202 where the returned product is displayed is highlighted so as to be distinguishable from the other display areas 202. The highlighting is performed by, for example, adding a special color such as red or yellow. In FIG. 8, the highlighting is shown by hatching. Along with the display of the notification screen 100, an alert sound such as a chime or a sound similar to the message 101 may also be output from the interface 13.

[0087] Display items 102, 103 and facial image 105 are displayed based on the information registered in the "product name," "return position," "reason for deterioration," "appearance abnormality," and "facial image" items of the product in the deterioration management information of Fig. 6. Fig. 8 shows an example of a notification screen 100 when the target product is a frozen or refrigerated product and the reason for deterioration is that the target product's removal time has exceeded. In the case of a notification due to an appearance abnormality, the reason for deterioration, such as a fall, and the details of the appearance abnormality are displayed in display item 103. Image 104 is displayed by highlighting an image prepared in advance for each product shelf 2.

[0088] By referring to the notification screen 100, the store clerk 4 can determine which product has been returned to the shelf 2, its location, and the reason for the deterioration of the product. After determining these, the store clerk 4 refers to the display area 202 to which the product has been returned to check whether any of the products displayed in the display area 202 has experienced any abnormalities. In the example of FIG. 8, condensation can be observed on the surface of a frozen product or frozen product that has been removed for a long period of time. This allows the store clerk 4 to identify the target product from the products in the display area 202 and collect the target product as appropriate. The store clerk 4 then operates the OK button 106 on the notification screen 100. This closes the notification screen 100 and ends the notification process.

[0089] Furthermore, the store clerk 4 can identify the customer 3 who performed the action on the product from the face image 105. This allows the store clerk 4 to call the attention of the customer 3 if the product is found to have deteriorated upon actual inspection.

[0090] FIG. 9 is a diagram showing the structure of customer management information for managing a customer 3 who has caused a product to be possibly deteriorated.

[0091] The customer management information is stored in the storage unit 12 and is updated as needed. The customer management information is configured by linking together a face image, the total number of notifications, and the number of notifications for each reason for deterioration.

[0092] The "face image" is a face image of the customer 3 extracted from the captured image. This face image may be further linked with the feature values of the feature parameters extracted from the face image. The "total number of notifications" is the total number of times that notifications were issued for the customer 3 in step S113 of FIG. 7. The "number of times per reason for deterioration" is the number of times that the "total number of notifications" is allocated to each reason for deterioration. These reasons for deterioration correspond to the reasons for deterioration that can be set in the "reason for deterioration" field in FIG. 6. The total number of times allocated to each reason for deterioration equals the number in the "total number of notifications" field.

[0093] FIG. 10 is a flowchart showing the registration process of customer management information.

[0094] When the control unit 11 registers flag information indicating notification in the "Notification" field of the deterioration management information in Fig. 6 (S201), it compares the facial image linked to this flag information with all facial images registered in the customer management information in Fig. 9 (S202). Then, when the facial image linked to the currently registered flag information matches any of the facial images registered in the customer management information (S203: YES), the control unit 11 updates the "total number of notifications" and the "number of times per reason for deterioration" linked to the matching facial image (S204).

[0095] Specifically, the control unit 11 increments the number in the "total number of notifications" field by 1. Furthermore, the control unit 11 increments the number in the field corresponding to the "reason for deterioration" linked to the flag information in the deterioration management information, among the multiple "reason for deterioration" fields prepared as the "number of times of reason for deterioration." In this way, the control unit 11 ends the processing of FIG. 10.

[0096] If the facial image linked to the newly registered flag information does not match any of the facial images registered in the customer management information (S203: NO), the control unit 11 adds a new row to the customer management information and newly registers information about the customer 3 in the customer management information (S205). Specifically, the control unit 11 registers the facial image linked to the flag information in the "Facial Image" field of the newly added row, and registers 1 in the "Total Number of Notifications" field of this row. Furthermore, the control unit 11 registers 1 in the field corresponding to the "Reason for Deterioration" linked to the flag information in the deterioration management information, among multiple "Reason for Deterioration" fields prepared as the "Number of Reasons for Deterioration" field. In this way, the control unit 11 ends the processing of FIG. 10.

[0097] FIG. 11 is a flowchart showing a process for notifying the store clerk 4 of a customer whose number of actions that satisfy the estimation conditions exceeds a predetermined threshold.

[0098] The process of Fig. 11 is performed by the control unit 11 using the function of the deterioration notification unit 11c. When the control unit 11 detects that a customer 3 has arrived at the store from an image captured by the imaging unit 20 capturing an image of the entrance S13 of Fig. 2 (S301: YES), the control unit 11 extracts a facial image of the customer 3 from the image captured by the imaging unit 20, and compares the extracted facial image with all facial images in the customer management information of Fig. 9 (S302). If no facial image matching the extracted facial image is present in the customer management information (S303: NO), the control unit 11 ends the process of Fig. 11.

[0099] On the other hand, if a face image matching the currently extracted face image exists in the customer management information (S303: YES), the control unit 11 refers to the total number of notifications associated with this face image in the customer management information and determines whether this total number of notifications exceeds a predetermined threshold (S304). This threshold is set to, for example, about 5 times. The store manager may be able to set this threshold as desired.

[0100] If the total number of notifications is equal to or less than the threshold (S304: NO), the control unit 11 ends the process in Fig. 11. On the other hand, if the total number of notifications exceeds the threshold (S304: YES), the control unit 11 executes a store visit notification process to notify the store clerk 4 that the customer has visited the store (S305).

[0101] FIG. 12 is a diagram showing the configuration of a notification screen 110 displayed on the interface 13 of FIG. 4 by the store visit notification process.

[0102] The notification screen 110 includes a message 111 notifying that a customer 3 with a history of deteriorating merchandise has visited the store, a facial image 112 of the customer 3, a display item 113 displaying the total number of deteriorating acts, a display item 114 displaying the total number of deteriorating acts divided into each deteriorating act, and an OK button 115. Along with the display of the notification screen 110, an alert sound such as a chime or a sound similar to the message 111 may also be output from the interface 13.

[0103] 9, the facial image of the customer 3, the total number of alerts, and the number of alerts for each reason for deterioration are displayed in the facial image 112 and the display items 113 and 114. That is, the facial information in the customer management information that matches the facial image of the customer 3 extracted from the image captured by the imaging unit 20 that captures the entrance S13, and the total number of alerts and the number of alerts for each reason for deterioration that are linked to this facial information are reflected in the facial image 112 and the display items 113 and 114 on the notification screen 110.

[0104] By referring to the notification screen 110, the store clerk 4 can know that a customer 3 with a deterioration history has visited the store, and can also know the total number of times that product deterioration has occurred and the breakdown of the number of times for each reason for deterioration. This allows the store clerk 4 to monitor the purchasing behavior of the customer 3 more carefully and smoothly check whether or not there has been any behavior that causes product deterioration. The store clerk 4 then operates the OK button 115 on the notification screen 110. This closes the notification screen 110 and ends the store visit notification process.

[0105] <Effects of the embodiment> As shown in Figures 2 and 4, the information processing device 10 is equipped with an action identification unit 11a that identifies the action of a customer 3 removing a product from the product shelf 2 and then returning the product to the product shelf 2 based on an image captured by an imaging unit 20 that can capture an image of the product on the product shelf 2 and a customer 3 accessing the product; a deterioration determination unit 11b that determines whether the action satisfies predetermined inference conditions for inferring the possibility that the product has deteriorated based on the image captured by the imaging unit 20; and a deterioration notification unit 11c that executes notification processing to notify a store manager (such as a store clerk 4) of information regarding the product that was the subject of the action (display items 102, 103, image 104 in Figure 8) based on the inference conditions being satisfied.

[0106] According to this configuration, the action of a customer 3 taking out an item and then returning it can be identified based on captured images of the item on the shelf 2 and the customer 3 accessing the item, so this action can be identified for any item displayed on the shelf 2, regardless of whether the item has specific temperature characteristics, such as refrigerated or frozen items. Furthermore, whether the action satisfies predetermined inference conditions based on the captured images determines whether the target item may have deteriorated, so this determination can be made for any item displayed on the shelf 2. Then, when the inference conditions are satisfied, information about the target item that may have deteriorated is notified to the store manager (e.g., store clerk 4), so the store manager, such as store clerk 4, can be smoothly notified that there may be an abnormality in the item that was returned to the shelf 2 by the customer 3. This allows the manager (e.g., store clerk 4) to smoothly and quickly check the condition of the target item. As described above, the estimation conditions in step S111 of FIG. 7 include that the target product is a refrigerated or frozen product, and that the time between when the customer 3 takes the target product from the shelf 2 and when they return it to the shelf 2 exceeds a threshold time.

[0107] According to this configuration, if the target product is a refrigerated or frozen product, and the time from when it was taken out to when it was returned exceeds the threshold time, and there is a possibility that the target product has deteriorated, information about the target product is notified to a manager such as a store clerk 4. This allows the manager to smoothly check the condition of the target product, which may have deteriorated, such as melted, due to being exposed to a room temperature environment for a long period of time.

[0108] As shown in Figure 7, the deterioration determination unit 11b determines whether or not an abnormality has occurred in the appearance of the target product before it is taken out and after it is put back based on the captured image (S109), and the estimation conditions in step S111 include the occurrence of an abnormality in the appearance.

[0109] According to this configuration, if an abnormality occurs in the appearance of the target product before it is taken out and after it is returned, information about the target product is notified to a manager such as a store clerk, thereby enabling the manager to smoothly check whether the target product has deteriorated.

[0110] As shown in Figure 7, the action identification unit 11a acquires the location where the target product was returned from the captured image (S108), and as shown in Figure 8, the deterioration notification unit 11c executes a process to display a notification screen 100 including information indicating the location where the target product was returned (product shelf and returned location in display item 102, image 104) as information about the target product.

[0111] According to this configuration, a manager such as a store clerk can ascertain the location where a possibly deteriorated product has been returned by referring to the notification screen 100. Therefore, the manager can smoothly and quickly check the condition of the product.

[0112] As shown in Figure 7, the action identification unit 11a acquires a facial image of the customer 3 who took out the target product and returned it from the captured image (S103), and as shown in Figure 8, the deterioration notification unit 11c displays the facial image 105 of the customer 3 on the notification screen 100.

[0113] According to this configuration, a manager such as a store clerk 4 can see the face of the customer 3 who performed the action that satisfies the estimation condition from the notification screen 100. Therefore, after checking the condition of the target product, the manager can ask the customer 3 who performed the action to take caution.

[0114] As shown in FIGS. 11 and 12, the deterioration notifying unit 11c further executes a process (S302 to S305) of displaying a face image 112 of a customer whose number of actions satisfying the estimation condition has exceeded a predetermined threshold.

[0115] According to this configuration, a manager such as a store clerk 4 can identify a customer 3 who repeatedly performs actions that satisfy the inferred conditions. Therefore, when this customer 3 visits the store, the manager can take measures such as strengthening monitoring of the customer 3.

[0116] <Change example 1> In the above embodiment, for products other than frozen and refrigerated products, the condition for inferring deterioration was that the product's appearance had become abnormal due to a deterioration event such as being dropped. However, if the damage to the product caused by a deterioration event such as being dropped is not visible in the captured image after the product is returned, the appearance abnormality cannot be detected from the captured image. In this case, the determination in step S111 of FIG. 7 is not YES, and the notification process in step S113 is not performed. As a result, the store clerk 4 cannot confirm that the product has become abnormal due to a drop or other reason, and the product cannot be quickly and smoothly collected.

[0117] To address this issue, in modified example 1, instead of the presumed condition that the product had an abnormal appearance when it was taken out and when it was returned, the presumed condition for deterioration is that some reason for deterioration has occurred.

[0118] In this case, the flowchart of Fig. 7 is changed to the flowchart of Fig. 13. Steps S109 and S110 relating to the determination and registration of visual abnormality in the flowchart of Fig. 7 are omitted in the flowchart of Fig. 13. Accordingly, the "visual abnormality" item is also omitted from the deterioration management information of Fig. 6. Furthermore, with regard to the estimation condition in step S111, of the above-mentioned multiple estimation conditions, the estimation condition "that a visual abnormality has occurred" is changed to the estimation condition "that a reason for deterioration has occurred." The other processing in the flowchart of Fig. 13 is the same as the processing of the corresponding steps in Fig. 7.

[0119] <Effects of Change Example 1> As shown in FIG. 13, the deterioration determination unit 11b determines whether a deterioration event has occurred in the target product based on the captured image (S105), and the estimation conditions in step S111 include that a deterioration event has occurred in the target product.

[0120] According to this configuration, information about products that may have been damaged or broken due to a deterioration cause such as a drop is notified to a manager such as a store clerk, allowing the manager to smoothly check whether the target products have been damaged or broken due to a deterioration cause.

[0121] <Change example 2> In the above embodiment, the deterioration management information of FIG. 6 was used in the process of notifying that the returned product had deteriorated, but this deterioration management information may also be used for other purposes.

[0122] For example, the deterioration management information may be used to analyze the purchasing behavior of customers 3. More specifically, facial images registered in the deterioration management information may be analyzed to obtain the age and gender of each customer 3, and an analysis may be performed to determine, for each age or gender, what type or category of product is likely to be removed from the shelf 2 and then returned to the shelf 2. In this case, the amount of time it takes for a product to be removed and returned may also be analyzed. Furthermore, by linking a date to each item of the deterioration management information, the above analysis may be performed for each day of the week, month, or season. Furthermore, the deterioration management information may be used to analyze which products of the same type are likely to be ultimately purchased by the same customer 3 based on their purchasing behavior after returning the product. Such an analysis may be performed for each store or region, or may be performed to obtain general purchasing trends regardless of store or region.

[0123] In this way, the deterioration management information is used to analyze the purchasing behavior of the customer 3, thereby providing useful analytical information to stores and companies. Such analysis may be performed by an analysis server connected to the network 40. In this case, the information processing device 10 may transmit the deterioration management information to be erased as metadata to the analysis server before erasing the deterioration management information shown in FIG. 6.

[0124] <Other change examples> In the above embodiment, the facial image of the customer captured when the product was removed was registered in the "Facial Image" field of the deterioration management information in Fig. 6. However, if the facial image of the customer cannot be extracted from the captured image when the product was removed, a facial image obtained from the captured image during subsequent behavior tracking may be registered in the "Facial Image" field. In this case, behavior tracking may be performed based on appearance information, such as the color of clothing of customer 3, that can be obtained from the captured image.

[0125] 6 includes the facial image 105 of the customer 3, the facial image 105 may be omitted from the notification screen 100. In this case, the store clerk 4 cannot warn the customer 3 who has committed a deteriorating act, but, as in the above embodiment, the store clerk 4 can quickly check the condition of the possibly deteriorated product from the display items 102, 103 and the image 104, and can smoothly take action such as recalling the product.

[0126] In this case, the "Facial Image" item may be omitted from the deterioration management information in Fig. 6. In this case, the control unit 11 may track the behavior of the customer 3 after picking up the product based on appearance information such as the color of the customer 3's clothes that can be acquired from the captured image.

[0127] 11 is executed in response to a customer 3's visit to the store, and the notification screen 110 of FIG. 12 is displayed, but the trigger for displaying the notification screen 110 is not limited to this. For example, the store clerk 4 may input an execution command from the interface 13 at any timing, so that a facial image of the customer 3 in the store is extracted from an image captured by any of the image capturing units 20, and the processing from step S302 onward in FIG. 11 is performed.

[0128] Alternatively, the store clerk 4 may input an execution command via the interface 13 at any timing, which may search the customer management information for facial images of customers 3 whose total number of notifications exceeds a threshold, and display a list of the facial images of these customers 3. In this case, the store clerk 4 may input to select one of the facial images displayed in the list, and the notification screen 110 of Fig. 12 may be displayed for the selected facial image.

[0129] Furthermore, the configuration of the product shelves 2 and the layout of the store S1 shown in the above embodiment are merely examples, and other configurations and layouts may be used. Similarly, the notification screens 100, 110 are not limited to the configurations shown in Figures 8 and 12, and may be screens with other configurations. Furthermore, as described above, the notification screens 100, 110 may be displayed on the mobile terminal 50 together with the information processing device 10, or the notification screens 100, 110 may be displayed on only one of the information processing device 10 and the mobile terminal 50.

[0130] In the above embodiment, the position of the display area 202 from which the product was removed was obtained by analyzing the captured image of the product shelf 2 on which the product was displayed, and the product variety and item were obtained from the obtained position and the management information in Figures 5(a) and (b), but the method of obtaining the product variety and item is not limited to this. For example, management information linking each product image with the product variety and item may be stored in advance in the storage unit 12, the product image in the management information that matches the product image removed from the product shelf 2 may be identified, and the product variety and item linked to the identified product image may be obtained as the product variety and item of the removed product.

[0131] Furthermore, the layout and items of the notification screens 100, 110 may differ between the information processing device 10 and the mobile terminal 50. For example, the notification screen 100 displayed on the mobile terminal 50 may omit the face image 105 or the image 104 from the items in Fig. 8, or the display item 103 may be omitted.

[0132] In addition, the embodiments of the present invention can be modified as appropriate within the scope of the claims. [Explanation of symbols]

[0133] 1. Product monitoring system 2 Product shelves 10. Information processing equipment 11 Control section 11a Act Identifier 11b Deterioration determination section 11c Deterioration notification section 100, 110 notification screen 105, 112 Facial images

Claims

1. an action recognition unit that recognizes an action of a customer returning a product to the product shelf after removing the product from the product shelf based on an image captured by an imaging unit that can capture an image of the product on the product shelf and a customer accessing the product; a deterioration determination unit that determines whether the action satisfies a predetermined estimation condition for estimating the possibility that deterioration has occurred in the product based on the image captured by the imaging unit; and a deterioration notification unit that executes a notification process to notify a store manager of information related to the target product that has been the target of the action based on the satisfaction of the estimation condition.

1. An information processing device comprising:

2. 2. The information processing device according to claim 1, The estimation conditions include that the target product is a refrigerated product or a frozen product, and the time from when the customer takes the target product off the shelf until when the customer puts the target product back on the shelf exceeds a threshold time.

1. An information processing device comprising:

3. 2. The information processing device according to claim 1, The deterioration determination unit determines whether a deterioration event has occurred in the target product based on the captured image, The presumed conditions include that the target product has deteriorated.

1. An information processing device comprising:

4. 2. The information processing device according to claim 1, The deterioration determination unit determines whether or not an abnormality has occurred in the appearance of the target product before it is taken out and after it is put back based on the captured image, The estimation condition includes occurrence of an abnormality in the appearance.

1. An information processing device comprising:

5. 2. The information processing device according to claim 1, The action identification unit acquires the position where the target product was returned from the captured image, the deterioration notification unit executes a process of displaying a notification screen including information indicating the position as information related to the target product.

1. An information processing device comprising:

6. 6. The information processing device according to claim 5, the behavior identification unit acquires a facial image of the customer who performed the behavior from the captured image; the deterioration notification unit displays the facial image on the notification screen, 1. An information processing device comprising:

7. 7. The information processing device according to claim 6, the deterioration notification unit further executes a process of displaying a face image of a customer whose number of times of the behavior that satisfies the estimation condition exceeds a predetermined threshold.

1. An information processing device comprising:

8. An information processing device according to any one of claims 1 to 7; The imaging unit, A merchandise surveillance system characterized by:

9. A program that causes a control unit of an information processing device that can communicate with an imaging unit that can capture images of products on a product shelf and customers accessing the products to execute the functions of each of the action identification unit, deterioration determination unit, and deterioration notification unit described in any one of claims 1 to 7.

Images