Fraud detection device, fraud detection system, fraud detection method, and fraud detection program

Abstract

To efficiently detect a fraudulent act during checkout for commodities at a self-checkout machine.SOLUTION: A camera 10 captures an image of a customer picking a commodity, and transmits the image to a management device 40 (S1). The management device 40 specifies the picked commodity from the received image (S2), and stores the specified commodity into to-be-purchased commodity data in association with a basket ID assigned to each basket. When the customer checks out a self-checkout machine 50, the self-checkout machine 50 transmits data of on a settled commodity to the management device 40 (S3). The management device 40 stores the received data of the commodity into checkout commodity data in association with the basket ID, and determines consistency between the to-be-purchased commodity and the checkout commodity (S4). When the result of the consistency determination indicates inconsistency, the result of inconsistency is transmitted to a clerk terminal 30 to notify a clerk (S5).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a fraud detection device, a fraud detection system, a fraud detection method, and a fraud detection program that can efficiently detect fraudulent behavior during product checkout at a self-checkout register. [Background technology]

[0002] In recent years, an increasing number of stores have introduced self-checkout registers, which allow shoppers to input the items they wish to purchase and pay for them themselves. Such self-checkout registers are also called self-checkout devices, self-checkout product registration terminals, or self-checkout terminals.

[0003] At such self-checkout machines, shoppers may engage in fraudulent behavior known as "register skipping." Specifically, shoppers may intentionally or accidentally not scan the items they intend to purchase, or may scan cheaper items instead of the items they actually intended to purchase.

[0004] For this reason, conventional technologies for preventing skipping checkouts are known. For example, Patent Document 1 discloses a fraud prevention system that prevents skipping by capturing images of items being added to a basket with a camera, identifying the items using a machine learning model, and then performing checkout. This fraud prevention system monitors the behavior of shoppers with in-store cameras, and uses this behavioral information to check whether the scan results are correct, thereby detecting fraud, and discloses technology for managing information about items being added to the basket using a smart cart or the shopper's smartphone. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-135620 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the system described in Patent Document 1 requires the management and processing of complex data, since it is necessary to capture the behavioral information of each customer who visits the store with an in-store camera, or to manage the product information that the customer is about to put in the basket using a smart cart or the customer's smartphone. Furthermore, in order to realize the system described in Patent Document 1, new equipment must be installed in the store, which incurs installation costs.

[0007] The present invention has been made to solve the problems of the above-mentioned conventional technology, and aims to provide a fraud detection device, a fraud detection system, a fraud detection method, and a fraud detection program that can efficiently detect fraudulent behavior during product checkout at a self-checkout register. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention is a fraud detection device for a self-service checkout system in which customers themselves perform a scanning operation to register products, and is characterized by comprising: an image acquisition unit that acquires a product pick image that is an image of a product that is installed in a store and used to pick products and that is captured in a recognizable manner, the image recognizable ID information assigned to a container used to pick products, and the product that is picked into the container; a memory unit that stores the product identified by the product pick image in association with the identification information of the container; a registered product identification unit that identifies the product registered in the self-service checkout system; a product confirmation unit that confirms whether the product identified by the registered product identification unit matches the product stored in association with the identification information of the container in which the product was stored; and an alarm unit that issues an alert if the product confirmation unit confirms that there is no match.

[0009] In the present invention, the container is a shopping basket or a shopping cart.

[0010] In the present invention, the identification information of the container is any one of number information, pattern information, and code information that uniquely identifies the container.

[0011] In addition, in the above invention, the present invention is characterized in that the registered product identification unit acquires registered product information from the self-checkout system via an interface.

[0012] In addition, in the above invention, the present invention is characterized in that the image acquisition unit also acquires a screen image when a product is registered in the self-service checkout system, and the registered product identification unit identifies the product information by performing character recognition on the screen image.

[0013] Furthermore, in the above invention, the present invention is characterized in that, when there is no consistency, the product confirmation unit extracts the inconsistent part, and the notification unit notifies the user along with the extraction results extracted by the product confirmation unit.

[0014] Furthermore, in the above invention, the present invention is characterized in that, if any of the products stored in association with the identification information of the container is not included among the products identified by the registered product identification unit, the product confirmation unit confirms that a product has not been registered, and the notification unit notifies that the product has not been registered.

[0015] In addition, in the above invention, the present invention is characterized in that when the product confirmation unit confirms that the product identified by the product pick image does not match the product identified by the registered product identification unit, the notification unit notifies that a product different from the actual product has been registered.

[0016] In the present invention, the storage unit erases the stored information when the product confirmation unit confirms that the information matches.

[0017] Furthermore, in the above invention, the present invention is characterized in that the image acquisition unit acquires a facial image of the customer along with the product pick image, the memory unit stores the facial image of the customer to whom the notification unit has made a notification in association with the number of notifications, and the notification unit executes a process to display the facial image of the customer to whom the number of notifications has exceeded a predetermined threshold.

[0018] The present invention also provides a fraud detection system for a self-checkout system in which customers themselves perform a scanning operation to register products, and is characterized by comprising: an image acquisition unit that acquires a product pick image that captures a recognizable image of image-recognizable identification information assigned to a container installed in a store and used when picking products, and the product picked from the container; a memory unit that stores product information of the product identified by the product pick image in association with the identification information of the container; a registered product identification unit that identifies product information of the product registered in the self-checkout system; a product confirmation unit that confirms whether the product information of the product identified by the registered product identification unit matches the product information of the product stored in association with the identification information of the container in which the product was stored; and an alarm unit that issues an alert if the product confirmation unit confirms that there is no match.

[0019] The present invention also provides a fraud detection method for a self-checkout system in which customers themselves perform a scanning operation to register products, and is characterized by including an image acquisition process for acquiring a product pick image in which image-recognizable identification information assigned to a container installed in a store and used when picking products and the product to be picked into the container are captured in a recognizable manner; a storage process for linking and storing product information of the product identified by the product pick image with the identification information of the container; a registered product identification process for identifying product information of the product registered in the self-checkout system; a product confirmation process for confirming whether the product information of the product identified in the registered product identification process matches the product information of the product stored in association with the identification information of the container in which the product was stored; and a notification process for issuing an alert if a mismatch is confirmed in the product confirmation process.

[0020] The present invention also provides a fraud detection program executed by a fraud detection device in a self-service checkout system in which customers themselves perform a scanning operation to register products, and includes an image acquisition procedure for acquiring a product pick image in which image-recognizable identification information assigned to a container installed in a store and used when picking products and the product picked into the container are recognizable; a storage procedure for linking and storing product information of the product identified by the product pick image with the identification information of the container; a registered product identification procedure for identifying product information of a product registered in the self-service checkout system; a product confirmation procedure for confirming whether the product information of the product identified in the registered product identification procedure matches the product information of the product stored in association with the identification information of the container in which the product was stored; and a notification procedure for issuing an alert if a mismatch is confirmed in the product confirmation procedure. [Effects of the Invention]

[0021] According to the present invention, fraudulent behavior during product checkout at a self-checkout can be efficiently detected. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an outline of a fraud detection system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating the system configuration of the fraud detection system according to the first embodiment. [Figure 3] FIG. 3 is a functional block diagram showing the configuration of the management device shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of the product management data and the product shelf data shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of the purchase candidate product data and the payment product data shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of the report history data shown in FIG. [Figure 7] FIG. 7 is a diagram illustrating an example of a display on the store clerk terminal according to the first embodiment. [Figure 8] FIG. 8 is a flowchart (part 1) illustrating a procedure for fraud detection processing according to the first embodiment. [Figure 9] FIG. 9 is a flowchart (part 2) illustrating the procedure of the fraud detection process according to the first embodiment. [Figure 10] FIG. 10 is a flowchart (part 3) illustrating the procedure of the fraud detection process according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a hardware configuration according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a display on a store clerk terminal according to a modified example. [Figure 13] FIG. 13 is a diagram (part 1) illustrating an outline of the fraud detection system according to the second embodiment. [Figure 14] FIG. 14 is a diagram (part 2) illustrating an outline of the fraud detection system according to the second embodiment. [Figure 15] FIG. 15 is a functional block diagram illustrating the configuration of a management device according to the second embodiment. [Figure 16] FIG. 16 is a diagram showing an example of the purchase candidate product data shown in FIG. [Figure 17] FIG. 17 is a diagram showing an example of the pick product data shown in FIG. [Figure 18] FIG. 18 is a diagram illustrating an example of correction of purchase candidate product data according to the second embodiment. [Figure 19] FIG. 19 is a flowchart illustrating a processing procedure for correcting purchase candidate product data according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] [Embodiment 1] Hereinafter, embodiments of a fraud detection device, a fraud detection system, a fraud detection method, and a fraud detection program will be described in detail with reference to the accompanying drawings.

[0024] <Outline of fraud detection system according to embodiment 1> First, an outline of the fraud detection system according to the present embodiment 1 will be described. Fig. 1 is an explanatory diagram showing an outline of the fraud detection system according to the present embodiment 1.

[0025] As shown in Figure 1(a), in the fraud detection system, a camera 10 captures video of a shopper picking up an item and transmits the video to a management device 40 (S1). The management device 40 identifies the picked item from the received video (S2), associates it with the basket ID attached to each basket, and stores it in the purchase intention item data.

[0026] As shown in FIG. 1(b), when a shopper pays at the self-checkout register 50, the self-checkout register 50 sends data of the paid items to the management device 40 (S3). The management device 40 stores the received item data in the payment item data in association with the basket ID, and determines whether the items to be purchased match the paid items (S4).

[0027] Specifically, the system compares the product information associated with the same basket ID in the purchase candidate data and the checkout product data to determine whether they match. For example, if the product information in the purchase candidate data with basket ID "123" is "3 tomatoes, 2 beef," and the product information in the checkout product data is "3 tomatoes, 1 beef," it determines that they are inconsistent.

[0028] If the result of the consistency check is that there is a mismatch, the store clerk terminal 30 is notified of the mismatch, and the store clerk is notified (S5).

[0029] In this way, the fraud detection system according to this embodiment 1 is configured to identify the items a shopper intends to purchase from video of the shopper picking out the items, determine whether they match the items being checked out at the self-checkout, and alert a store clerk if there is a mismatch, thereby enabling fraudulent behavior during checkout at the self-checkout to be detected efficiently.

[0030] <System Configuration of the Fraud Detection System According to the First Embodiment> Next, a description will be given of the system configuration of the fraud detection system according to the present embodiment 1. Fig. 2 is a diagram showing the system configuration of the fraud detection system according to the present embodiment 1.

[0031] 2, a management device 40 installed in an office or the like of a store is communicably connected to a camera 10, a wireless router 20, and a self-checkout register 50 via a communication line. The wireless router 20 is connected to a store clerk terminal 30 via short-range wireless communication such as Wi-Fi (registered trademark).

[0032] Camera 10 is an imaging device installed in a predetermined location in the store so that it can capture images of shopping in the store and checkout at self-checkout 50. Camera 10 transmits the captured images to management device 40.

[0033] The clerk terminal 30 is a terminal such as a tablet carried by a clerk. When the clerk terminal 30 receives a customer warning from the management device 40, it displays a customer warning screen, and when it receives a fraudulent payment warning, it displays a fraudulent payment warning screen.

[0034] The self-checkout register 50 is a device that allows shoppers to pay for products themselves. When the self-checkout register 50 receives a payment product data request from the management device 40, it transmits the payment product data to the management device 40.

[0035] The management device 40 is a device that detects fraudulent transactions related to the payment of merchandise. When the management device 40 receives data related to products, it stores the received data in product management data, and when it receives product display information for each product shelf, it stores the received product display information for each product shelf in product shelf data.

[0036] Furthermore, the management device 40 determines that the face image of the shopper shown in the video data acquired from the camera 10 is the same person as the face image stored in the report history data, and if the number of reports in the past is three or more and no report has been made on the day, it notifies the store clerk terminal 30 of a warning about a customer requiring caution, including this face image.

[0037] The management device 40 also reads a number from a number tag attached to a shopping basket or shopping cart captured in the video data acquired from the camera 10, and acquires this number as a basket ID. It then acquires a facial image of a shopper holding a shopping basket or pushing a shopping cart corresponding to this basket ID, and stores the facial image in association with the basket ID in the purchase candidate product data.

[0038] Furthermore, if the management device 40 detects the action of taking a product from a product shelf and putting it into a shopping basket or shopping cart in the video data acquired from the camera 10, it identifies the relevant product using the position where the product was taken from the product shelf, the product management data, and the product shelf data.The management device 40 then acquires the basket ID of the shopping basket or shopping cart into which the product has been placed, and stores the identified product name in association with this basket ID in the purchase candidate product data.

[0039] The management device 40 also uses the video data captured by the camera 10 of the checkout status at the self-checkout 50 to identify the register ID of the self-checkout 50 where the shopper made the payment, and sends a request for payment item data to the self-checkout 50 corresponding to this register ID. When the management device 40 receives payment item data from the self-checkout 50, it associates the data with the basket ID and register ID and stores it as payment item data.

[0040] Furthermore, when the payment item data is updated, the management device 40 identifies the basket ID of the updated data and determines whether the purchase candidate item data corresponding to this basket ID is consistent with the updated payment item data. If it determines that there is an inconsistency, it notifies the store clerk terminal 30 of a fraudulent payment alert and updates the number of reports in the report history data by adding 1. If it determines that there is a consistency, it deletes the data related to the consistency determination from the purchase candidate item data and the payment item data, and updates them.

[0041] <Configuration of management device 40> Next, a description will be given of the configuration of the management device 40 shown in Fig. 2. Fig. 3 is a functional block diagram showing the configuration of the management device 40 shown in Fig. 2. As shown in Fig. 3, the management device 40 is connected to a display unit 41 and an input unit 42, and has a communication unit 44, a storage unit 45, and a control unit 46.

[0042] The display unit 41 is a display device such as a liquid crystal panel or a display device. The input unit 42 is an input device such as a keyboard or a mouse. The communication unit 44 is an interface unit for communicating data with the camera 10, the store clerk terminal 30, and the self-checkout 50 via a communication line.

[0043] The storage unit 45 is a storage device such as a hard disk drive or nonvolatile memory, and stores product management data 45a, product shelf data 45b, face image data 45c, purchase candidate product data 45d, payment product data 45e, and report history data 45f.

[0044] The product management data 45a is data showing information about products. The product shelf data 45b is data showing product numbers displayed on product shelves and showing shelf allocation information. The face image data 45c is data showing a face image of a shopper.

[0045] The intended purchase item data 45d is data showing product information of items that a shopper has picked up and put into a basket. The payment item data 45e is data showing product information of items that have been paid for at the self-checkout 50. The report history data 45f is data that stores facial information of customers suspected of committing fraud when fraud is determined to have occurred in the payment for items, and stores the number of times each shopper has been reported.

[0046] The control unit 46 is a control unit that performs overall control of the management device 40, and includes a product management unit 46a, a product shelf management unit 46b, a video acquisition unit 46c, a facial image acquisition unit 46d, a facial image determination unit 46e, a basket ID acquisition unit 46f, a human movement detection unit 46g, a purchase candidate identification unit 46h, a cash register ID identification unit 46i, a registered product identification unit 46j, a product determination unit 46k, an alarm unit 46l, and a data deletion unit 46m. In practice, by loading these programs into a CPU (Central Processing Unit) and executing them, the product management unit 46a, the product shelf management unit 46b, the video acquisition unit 46c, the facial image acquisition unit 46d, the facial image determination unit 46e, the basket ID acquisition unit 46f, a human movement detection unit 46g, a purchase candidate identification unit 46h, a cash register ID identification unit 46i, a registered product identification unit 46j, a product determination unit 46k, an alarm unit 46l, and a data deletion unit 46m are caused to execute processes corresponding to the product management unit 46a, the product shelf management unit 46b, the video acquisition unit 46c, the facial image acquisition unit 46d, the facial image determination unit 46e, the basket ID acquisition unit 46f, a human movement detection unit 46g, a purchase candidate identification unit 46h, a cash register ID identification unit 46i, a registered product identification unit 46j, a product determination unit 46k, an alarm unit 46l, and a data deletion unit 46m.

[0047] The product management unit 46a is a processing unit that manages the product management data 45a. When the product management unit 46a receives data related to products from the input unit 42, the product management unit 46a stores the received data in the product management data 45a.

[0048] The product shelf management unit 46b is a processing unit that manages the product shelf data 45b. When the product shelf management unit 46b receives product display information for each product shelf from the input unit 42, it stores the received product display information for each product shelf in the product shelf data 45b. The product display information for each product shelf includes product numbers for each upper, lower, and left and right shelf division of the product shelf. For example, the upper and lower shelf divisions are upper, middle, and lower, and the left and right shelf divisions are left shelf, middle shelf, and right shelf.

[0049] The video acquisition unit 46c is a processing unit that acquires video images captured by the cameras 10. The video acquisition unit 46c acquires video data transmitted from all of the cameras 10 installed in the store via a communication line.

[0050] The facial image acquisition unit 46d is a processing unit that acquires a facial image of the shopper. The facial image acquisition unit 46d recognizes the facial image of the shopper that appears in the video data acquired by the video acquisition unit 46c, acquires this facial image, and stores it in the facial image data 45c.

[0051] The face image determination unit 46e is a processing unit that determines whether the face image acquired by the face image acquisition unit 46d is the same person as the face image stored in the report history data 45f. If the face image determination unit 46e determines that the face image acquired by the face image acquisition unit 46d is the same person as the face image stored in the report history data 45f, it passes the watchful customer information including the face image of the corresponding report history data 45f to the notification unit 46l.

[0052] The basket ID acquisition unit 46f is a processing unit that acquires basket IDs assigned to shopping baskets and shopping carts. The basket ID acquisition unit 46f reads a number from a number tag attached to a shopping basket or shopping cart that appears in the video data acquired by the video acquisition unit 46c, and acquires this number as a basket ID.

[0053] Furthermore, when the basket ID acquisition unit 46f acquires a basket ID, the facial image acquisition unit 46d acquires a facial image of a shopper holding a shopping basket corresponding to the basket ID or a shopper pushing a shopping cart, and stores the facial image in association with the basket ID in the purchase candidate product data 45d.

[0054] The human action detection unit 46g is a processing unit that detects the picking action of a shopper. The human action detection unit 46g detects the action of picking a prize from the action of the shopper captured in the video data acquired by the video acquisition unit 46c. Specifically, the human action detection unit 46g detects the action of taking a product from a shelf, the action of putting a product back, and the action of putting a product into a shopping basket or shopping cart, as picking actions.

[0055] The intended purchase product identification unit 46h is a processing unit that identifies products placed in a shopping basket or shopping cart. When the intended purchase product identification unit 46h detects the action of removing a product from a shelf and placing the product in a shopping basket or shopping cart in the picking action detected by the human action detection unit 46g, the intended purchase product identification unit 46h identifies the corresponding product using the position where the product was removed from the shelf, the product management data 45a, and the product shelf data 45b. Then, the basket ID of the shopping basket or shopping cart into which the product is placed is obtained by the basket ID acquisition unit 46f, and the identified product name associated with this basket ID is stored in the intended purchase product data 45d.

[0056] The register ID identification unit 46i is a processing unit that identifies the register ID used when a shopper settles at a self-checkout register 50. The register ID identification unit 46i identifies the register ID of the self-checkout register 50 at which the shopper settled using video data that captures the settlement status at the self-checkout register 50, from the video data acquired by the video acquisition unit 46c.

[0057] The registered product identification unit 46j is a processing unit that acquires payment product data. The registered product identification unit 46j notifies a payment product data request to the self-checkout register 50 corresponding to the register ID identified by the register ID identification unit 46i. At this time, the basket ID acquisition unit 46f acquires the basket ID of the shopping basket or shopping cart used for payment at this self-checkout register 50. Then, when payment product data is received from the self-checkout register 50, it is stored in payment product data 45e in association with the basket ID and register ID.

[0058] The product determination unit 46k is a processing unit that determines whether or not there is any fraud in the payment of products. When the payment product data 45e is updated by the registered product identification unit 46j, the product determination unit 46k identifies the basket ID of the updated data and extracts the purchase candidate product data corresponding to this basket ID from the purchase candidate product data 45d. The product determination unit 46k then compares the product information in the updated payment product data 45e with the product information in the extracted purchase candidate product data 45d to determine whether they are consistent with each other. If an inconsistency is determined, the product determination unit 46k passes fraudulent payment information including the cash register ID, basket ID, amount in the payment product data 45e, and amount in the purchase candidate product data 45d, as well as the facial image of the extracted purchase candidate product data 45d, to the notification unit 46l. If a consistency is determined, the product determination unit 46k passes payment completion information to the data deletion unit 46m.

[0059] The notification unit 46l is a processing unit that notifies the store clerk terminal 30 of a customer alert and a fraudulent payment alert. When the notification unit 46l receives customer alert information from the face image determination unit 46e, it identifies the number of reports and the report status corresponding to the face image included in the customer alert information from the report history data 45f. If the identified number of reports is three or more and the report status is "not yet", the notification unit 46l notifies the store clerk terminal 30 of a customer alert including the face image, and updates the report status in the report history data 45f to "completed".

[0060] In addition, when the notification unit 46l receives fraudulent payment information from the product evaluation unit 46k, it notifies the store clerk terminal 30 of this fraudulent payment information as a fraudulent payment alarm, and updates the number of reports in the report history data 45f corresponding to the face image received from the product evaluation unit 46k by adding 1.

[0061] The data deletion unit 46m is a processing unit that deletes the corresponding data when the purchase candidate product and the payment candidate product are matched. Upon receiving payment completion information from the product determination unit 46k, the data deletion unit 46m deletes the data related to the determination by the product determination unit 46k from the purchase candidate product data 45d and the payment candidate product data 45e, and updates them.

[0062] Next, an example of data stored in the storage unit 45 of the management device 40 shown in Fig. 3 will be described. Figs. 4 to 6 are diagrams showing examples of the product management data 45a, product shelf data 45b, purchase candidate product data 45d, payment product data 45e, and report history data 45f shown in Fig. 3.

[0063] The product management data 45a shown in Figure 4(a) associates the product number "11001" with the product category "daily necessities" and the product name "Kireider, powerful detergent," and associates the product number "11002" with the product category "daily necessities" and the product name "Fuwatter, foaming detergent."

[0064] The product shelf data 45b shown in Figure 4(b) associates the product shelf number "2101" and the upper / lower shelf classification "upper shelf" with the product number "11001" on the left shelf, the product number "11002" on the middle shelf, and the product number "11003" on the right shelf, and associates the product shelf number "2101" and the upper / lower shelf classification "middle shelf" with the product number "11013" on the left shelf, the product number "11015" on the middle shelf, and the product number "11019" on the right shelf.

[0065] Furthermore, the product shelf data 45b associates the product shelf number "2101" and the upper / lower shelf classification "lower shelf" with the product number "11022" on the left shelf, the product number "11027" on the middle shelf, and the product number "11033" on the right shelf, and associates the product shelf number "2102" and the upper / lower shelf classification "upper shelf" with the product number "12005" on the left shelf, the product number "12011" on the middle shelf, and the product number "12026" on the right shelf.

[0066] 5(a) associates the face image "012301.jpg" and the price "2050" yen with the basket ID "123." Furthermore, the product information associated with the basket ID "123" is the product name "tomato," the quantity "3," the cashier status "yet to be paid," the product name "cabbage," the quantity "1," the cashier status "yet to be paid," the product name "beef," the quantity "2," and the cashier status "yet to be paid."

[0067] Furthermore, the purchase intention product data 45d associates the face image "012401.jpg" and the price "1120" yen with the basket ID "456." Furthermore, the product information associated with the basket ID "456" is the product name "sauce," the quantity "1," the cashier status "yet," the product name "pepper," the quantity "3," the cashier status "yet," the product name "banana," the quantity "1," and the cashier status "yet."

[0068] 5(b), the payment product data 45e associates the basket ID "123" with a cash register ID of "001" and a price of "1890" yen. Furthermore, the product information associated with the basket ID "123" is the product name "tomato" and the quantity of "3", the product name "cabbage" and the quantity of "1", and the product name "beef" and the quantity of "1".

[0069] Additionally, the checkout product data 45e associates the basket ID "456" with a cash register ID of "004" and a purchase amount of "1120" yen. Furthermore, the product information associated with the basket ID "456" is the product name "sauce" and the quantity "1", the product name "pepper" and the quantity "3", and the product name "banana" and the quantity "1".

[0070] The reporting history data 45f shown in Figure 6 associates the facial image "010201.jpg" with a state in which the number of reports is "5" and the reporting status is "completed," and associates the facial image "011701.jpg" with a state in which the number of reports is "2" and the reporting status is "not yet."

[0071] <Example of display on the store clerk terminal 30 according to the first embodiment> Next, an example of a display on the clerk terminal 30 according to the present embodiment 1 will be described. Fig. 7 is a diagram showing an example of a display on the clerk terminal 30 according to the present embodiment 1. When the clerk terminal 30 receives a fraudulent payment warning from the management device 40, it displays a fraudulent payment warning screen, and when it receives a customer warning, it displays a customer warning screen.

[0072] As shown in FIG. 7(a), the fraudulent payment warning screen displays a message that fraud has occurred, the register ID and basket ID of the fraudulent activity, and the amounts of the items paid for and the items to be purchased.

[0073] For example, it displays "Fraud has occurred. Please check," "Self-register: No. 001, Basket ID: No. 123," and "Register total: 1,890 yen, Basket total: 2,050 yen." The register total is the total amount of the items paid for at the self-register 50, and the basket total is the total amount expected when the items placed in the shopping basket are paid for.

[0074] As shown in Figure 7(b), the customer warning screen displays the fact that a shopper with a history of fraudulent activities has visited the store, the number of reports of fraudulent activities, and an image of the shopper's face. For example, it displays "A customer with a history of fraudulent activities has visited the store!" and "5 reports."

[0075] <Processing Procedure for Fraud Detection According to the First Embodiment> Next, a description will be given of the procedure for fraud detection processing according to the present embodiment 1. Figures 8 to 10 are flowcharts showing the procedure for fraud detection processing according to the present embodiment 1.

[0076] 8, if the management device 40 acquires a basket ID from the video received from the camera 10 installed to capture images of the sales floor (step S101: Yes), it acquires a facial image of the shopper from the received video (step S102). If the management device 40 cannot acquire a facial image (step S102: No), it proceeds to step S105.

[0077] If a face image is acquired (step S102: Yes), the acquired face image is stored in the face image data 45c (step S103), and a caution customer warning notification process is performed (step S104).

[0078] In the caution passenger warning notification process, as shown in FIG. 10, if the acquired face image is not the same person as the face image in the report history data 45f (step S301: No), the process proceeds to step S105.

[0079] If the acquired image is the same person as the face image in the report history data 45f (step S301: Yes), the number of reports of the person in this face image is checked in the report history data 45f (step S302). If the number of reports is less than 3 (step S302: No), proceed to step S105.

[0080] If the number of reports is 3 or more (step S302: Yes), the system checks the report history data 45f to see if the person in the face image has been reported (step S303). If the report status in the report history data 45f is "reported" (step S303: Yes), the system proceeds to step S105.

[0081] If the report status in the report history data 45f is "Not yet" (step S303: No), a cautionary customer alert is sent to the store clerk terminal 30 (step S304), and the process proceeds to step S105. At this time, the report status in the report history data 45f is updated to "Completed".

[0082] In step S105 shown in FIG. 8, if picking of a product from a product shelf to a shopping basket or shopping cart is not detected (step S105: No), the process proceeds to step S107.

[0083] If a product is detected being picked from a shelf and put into a shopping basket or shopping cart (step S105: Yes), the location where the product was taken from the shelf, the product management data 45a, and the product shelf data 45b are used to identify the product in question, and the identified product name is stored in the purchase target product data 45d (step S106).

[0084] If the same basket ID as that acquired in step S101 continues to be acquired from the video received from the camera 10 in the same sales floor as when the basket ID was acquired in step S101 (step S107: Yes), the process proceeds to step S102. If the same basket ID has not been acquired (step S107: No), the process proceeds to step S108.

[0085] If the image received from the camera 10 installed to capture the self-checkout 50 does not contain the same basket ID as the basket ID acquired in step S101 (step S108: No), the process proceeds to step S101. If the same basket ID is acquired (step S108: Yes), the process proceeds to step S201.

[0086] The register ID of the self-checkout register 50 where the shopper made the payment is identified from the same image as that used in step S108 (step S201). Then, the payment item data is obtained from the self-checkout register 50 corresponding to this register ID (step S202) and stored in the payment item data 45e.

[0087] Information on the purchase candidate products in the purchase candidate product data 45d and the payment candidate products in the payment candidate product data 45e related to the basket ID acquired in step S108 is extracted and compared (step S203).

[0088] If the purchase candidate product and the payment candidate product match (step S203: Yes), the corresponding data in the purchase candidate product data 45d and payment candidate product data 45e are deleted (step S206), and the process ends.

[0089] If the product to be purchased and the product to be paid for do not match (step S203: No), A fraudulent payment alarm is sent to the store clerk terminal 30 (step S204), and the number of reports in the report history data 45f is updated by adding 1 (step S205), and the process ends.

[0090] <Example of hardware configuration according to the first embodiment> Next, a description will be given of the correspondence between the management device 40 of the fraud detection system according to the present embodiment and the main hardware configuration of a computer. Fig. 11 is a diagram showing an example of the hardware configuration according to the present embodiment.

[0091] Generally, a computer has a configuration in which a CPU 81, a ROM 82, a RAM 83, a non-volatile memory 84, etc. are connected via a bus 85. A hard disk drive may be provided instead of the non-volatile memory 84. For the sake of convenience of explanation, only the basic hardware configuration is shown.

[0092] Here, the ROM 82 or non-volatile memory 84 stores programs required to start the operating system (hereinafter simply referred to as "OS"), and the CPU 81 reads and executes the OS program from the ROM 82 or non-volatile memory 84 when the power is turned on.

[0093] On the other hand, various application programs executed on the OS are stored in non-volatile memory 84, and the CPU 81 executes the application programs while using RAM 83 as the main memory, thereby executing processes corresponding to the applications.

[0094] The fraud detection program of the management device 40 of the fraud detection system according to the first embodiment is also stored in the non-volatile memory 84 or the like, like other application programs, and the CPU 81 loads and executes this program. In the case of the management device 40 of the fraud detection system according to the first embodiment, the fraud detection program including routines corresponding to the product management unit 46a, product shelf management unit 46b, video acquisition unit 46c, facial image acquisition unit 46d, facial image determination unit 46e, basket ID acquisition unit 46f, human motion detection unit 46g, intended purchase product identification unit 46h, register ID identification unit 46i, registered product identification unit 46j, product determination unit 46k, notification unit 46l, and data deletion unit 46m shown in FIG. 3 is stored in the non-volatile memory 84 or the like. When the fraud detection program is loaded and executed by the CPU 81, fraud detection processes corresponding to the product management unit 46a, product shelf management unit 46b, video acquisition unit 46c, facial image acquisition unit 46d, facial image determination unit 46e, basket ID acquisition unit 46f, human movement detection unit 46g, intended purchase product identification unit 46h, cash register ID identification unit 46i, registered product identification unit 46j, product determination unit 46k, notification unit 46l and data deletion unit 46m are generated.

[0095] As described above, the fraud detection system according to this embodiment 1 is configured to identify the items a shopper intends to purchase from video of the shopper picking out the items, determine whether they match the items being checked out at the self-checkout, and alert a store clerk if there is a mismatch, thereby enabling fraudulent behavior during checkout at the self-checkout to be detected efficiently.

[0096] In the first embodiment, the basket ID is number information, but the present invention is not limited to this. The basket ID may be pattern information such as a logo, design, or pattern diagram, or code information such as a QR code (registered trademark).

[0097] In the first embodiment, the payment product data is acquired from the self-checkout register via a communication line. However, the present invention is not limited to this. The payment product data may be acquired by capturing an image of the payment product information displayed on the display unit of the self-checkout register with a camera and detecting text information related to the payment product from the image.

[0098] [Variations] In the above embodiment 1, a configuration was described in which a match between the items to be purchased placed in the shopping basket and the items to be paid for at the self-checkout is determined, and if there is a mismatch, the total amount of each item is notified to the store clerk, but the present invention is not limited to this.

[0099] Possible reasons for a mismatch between the product to be purchased and the product to be paid for include not scanning the product barcode at the self-checkout, leaving the product blank, or replacing the barcode of a different product. If a mismatch occurs between the product to be purchased and the product to be paid for, the system can be configured to notify the store clerk of the details corresponding to these reasons. Specifically, the system can be configured to notify the store clerk of the details of the mismatch, the details of the unpaid item, or the registered details of the different product.

[0100] In this variant, we will describe a fraud detection system that determines whether the items intended for purchase placed in the shopping basket match the items paid for at the self-checkout, and if there is a mismatch, notifies the store clerk of the details of the mismatch, the details of the unpaid items, or the registered details of the items that are different.

[0101] <Example of display on store clerk terminal 30 according to modified example> An example of a display on the clerk terminal 30 according to this modification will be described below. Fig. 12 is a diagram showing an example of a display on the clerk terminal 30 according to this modification.

[0102] As shown in Figure 12(a), when reporting the details of the inconsistency, the following is displayed: "An inconsistency has occurred. Please check.", "Self-checkout: No. 001, Basket ID: No. 123," and "Items paid at the register: Beef x 1, Items in the basket: Beef x 2." This display makes it possible to report the names of the inconsistent items and their quantities.

[0103] As shown in Figure 12(b), when reporting the details of unpaid items, the following is displayed: "There is a risk of items being left out. Please check," "Self-checkout: No. 001, basket ID: No. 123," and "The next item is unpaid. Beef x 1." This display can notify the user that there are unpaid items and that there is a risk of items being left out.

[0104] As shown in Figure 12(c), when reporting the registered details of a different product, the following is displayed: "There is a product discrepancy. Please check," "Self-checkout: No. 001, basket ID: No. 123," and "Register payment item: Chicken x 1, basket item: Beef x 1." This display can notify that a different product has been registered and that there is a risk of the barcode being replaced with a different product.

[0105] As described above, the fraud detection system of this modified example determines whether the items to be purchased placed in the shopping basket match the items to be paid for at the self-checkout, and if there is a mismatch, it notifies the store clerk of the details of the mismatch, the details of the unpaid items, or the registered details of the different items, thereby making it possible to efficiently detect fraudulent behavior when paying for items at the self-checkout.

[0106] [Embodiment 2] In the above-mentioned first embodiment, a configuration was described in which the items to be purchased are identified from a video of a single shopper picking out the items, and a match is determined with the items to be checked out at the self-checkout, and if there is a mismatch, a store clerk is notified. However, the present invention is not limited to this.

[0107] There are cases where multiple shoppers visit a store, and each shopper may pick their own products. Therefore, the challenge is to combine the products picked by multiple shoppers and identify them as a single intended purchase.

[0108] To solve this problem, multiple shoppers who come to the store can be associated with a single basket ID, which identifies the items picked by each shopper as a single item they plan to purchase.The system then determines whether the items match the items being checked out at the self-checkout and alerts the store clerk if there is a mismatch.

[0109] In this second embodiment, we will explain a fraud detection system that associates multiple shoppers who come to the store with a single basket ID, identifies the items picked by each shopper as a single item they plan to purchase, determines whether they match the items being checked out at the self-checkout, and alerts the store clerk if there is a mismatch.

[0110] <Outline of fraud detection system according to embodiment 2> An outline of the fraud detection system according to the second embodiment will be described below. Figures 13 and 14 are explanatory diagrams showing an outline of the fraud detection system according to the second embodiment.

[0111] 13, in the fraud detection system according to the second embodiment, the camera 10a transmits video of multiple shoppers visiting a store to the management device 100 (S11). The management device 100 associates facial images of the multiple shoppers from the received video with one basket ID (S12).

[0112] There are cases where one of multiple shoppers (for example, shopper B) puts a product in the shopping basket, but the product is unknown (S13). For example, if it is possible to identify that shopper A put three tomatoes in the shopping basket, but it is not possible to identify the product that shopper B put in, the product information in the intended purchase product data remains "three tomatoes."

[0113] On the other hand, the fraud detection system according to the second embodiment identifies the product that the shopper has picked from the product shelf. Specifically, as shown in Fig. 14(a), the camera 10b transmits a video of the shopper picking the product from the product shelf to the management device 100 (S14).

[0114] The management device 100 identifies the face image of the shopper and the picked product from the received video (S15) and stores them in the picked product data. For example, "one piece of gum" is stored in the product information of the picked product data.

[0115] As shown in Figure 14(b), when the product placed in the shopping basket is unknown, the fraud detection system according to the second embodiment identifies the product by searching for the same basket ID and face image from the purchase intention product data and the picked product data. For example, it identifies that the unknown product placed in basket ID "123" is "one piece of gum."

[0116] In this way, the fraud detection system of this embodiment 2 associates multiple shoppers who come to the store with a single basket ID, so that the items picked by each shopper can be identified as a single item to be purchased, and the system can determine whether the items match the items being checked out at the self-checkout register, and alert the store clerk if there is a mismatch.

[0117] <Configuration of management device 100 according to the second embodiment> Next, the configuration of the management device 100 according to the second embodiment will be described. Fig. 15 is a functional block diagram showing the configuration of the management device 100 according to the second embodiment. As shown in Fig. 15, the management device 100 is connected to a display unit 41 and an input unit 42, and has a communication unit 44, a storage unit 110, and a control unit 120. Note that a description of functional units similar to those of the management device 40 shown in Fig. 3 will be omitted.

[0118] The memory unit 110 is a storage device such as a hard disk drive or non-volatile memory, and stores product management data 45a, product shelf data 45b, face image data 45c, purchase target product data 111, checkout product data 45e, report history data 45f, and pick product data 112.

[0119] The purchase intention product data 111 is data showing product information of products that a shopper has picked up to put in a basket. The picked product data 112 is data showing product information of products that a shopper has picked up from a product shelf.

[0120] The control unit 120 is a control unit that performs overall control of the management device 100, and has a product management unit 46a, a product shelf management unit 46b, a video acquisition unit 46c, a face image acquisition unit 46d, a face image determination unit 46e, a basket ID acquisition unit 121, a person's movement detection unit 46g, a picked product identification unit 122, a product intended for purchase identification unit 123, a cash register ID identification unit 46i, a registered product identification unit 46j, a product determination unit 46k, an alarm unit 46l, and a data deletion unit 46m. In practice, by loading these programs into the CPU and executing them, the processes corresponding to the product management unit 46a, product shelf management unit 46b, video acquisition unit 46c, facial image acquisition unit 46d, facial image determination unit 46e, basket ID acquisition unit 121, human movement detection unit 46g, picked product identification unit 122, intended purchase product identification unit 123, cash register ID identification unit 46i, registered product identification unit 46j, product determination unit 46k, notification unit 46l and data deletion unit 46m will be executed.

[0121] The basket ID acquisition unit 121 is a processing unit that acquires basket IDs assigned to shopping baskets and shopping carts. The basket ID acquisition unit 121 reads a number from a number tag attached to a shopping basket or shopping cart that appears in the video data acquired by the video acquisition unit 46c, and acquires this number as a basket ID.

[0122] Furthermore, when the basket ID acquisition unit 121 acquires a basket ID, it acquires a facial image of a shopper holding a shopping basket corresponding to the basket ID or a shopper pushing a shopping cart (hereinafter referred to as a "shopper currently shopping") using the facial image acquisition unit 46d, and stores the facial image in the purchase candidate product data 111 in association with the basket ID.

[0123] Furthermore, if another shopper accompanies the shopping customer for a predetermined time or longer, the basket ID acquisition unit 121 acquires a facial image of this other shopper using the facial image acquisition unit 46d, associates the acquired facial image with the basket ID associated with the shopping customer, and stores the acquired facial image as a related facial image of the purchase candidate product data 111.

[0124] The pick product identification unit 122 is a processing unit that identifies products picked by a shopper from the product shelves. When the pick action detected by the human action detection unit 46g detects the action of removing a product from the product shelves, the pick product identification unit 122 identifies the corresponding product using the product's pick position on the product shelves, the product management data 45a, and the product shelf data 45b. Furthermore, the face image acquisition unit 46d acquires a facial image of the shopper, and identifies the basket ID corresponding to this facial image from the purchase intention product data 111. Then, the acquired facial image is associated with the identified basket ID, product, pick action (removal), and its time, and stored in the pick product data 112. At this time, the basket management status of the pick product data 112 is stored as "not yet."

[0125] Furthermore, if the pick action detected by the human action detection unit 46g is the action of returning an item to a product shelf, the pick product identification unit 122 identifies the corresponding item using the product's pick position on the product shelf, the product management data 45a, and the product shelf data 45b. Furthermore, the facial image of the shopper is acquired by the facial image acquisition unit 46d, and this facial image is stored and updated in association with the pick action (return) and time in the pick product data 112 corresponding to the identified item. At this time, the basket management status of the pick product data 112 is updated to "returned," and if an abnormality occurs, such as the identified item being dropped, the abnormality is stored in the abnormality field of the pick product data 112.

[0126] The purchase candidate product identification unit 123 is a processing unit that identifies products placed in a shopping basket or shopping cart. When the person's motion detection unit 46g detects a pick action in which a product is removed from a product shelf and placed in a shopping basket or shopping cart, the purchase candidate product identification unit 123 identifies the corresponding product using the position where the product was removed from the product shelf, the product management data 45a, and the product shelf data 45b. The purchase candidate product identification unit 123 then acquires the basket ID of the shopping basket or shopping cart into which the product has been placed using the basket ID acquisition unit 121, and stores the identified product name in association with this basket ID in the purchase candidate product data 111, and updates the basket management status of the pick product data 112 corresponding to this basket ID and product name to "completed."

[0127] Furthermore, if the human motion detection unit 46g detects a picking motion in which a product is placed in a shopping basket or shopping cart without actually removing the product from a shelf, the purchase candidate identification unit 123 acquires a facial image of the shopper who performed this picking motion using the facial image acquisition unit 46d, and identifies from the picked product data 112 the product whose basket management status is "not yet" in the product information corresponding to this facial image.The purchase candidate identification unit 123 then stores and updates the purchase candidate data 111 associated with the basket ID corresponding to the shopping basket or shopping cart into which the product has been placed.At this time, the basket management status of the product identified from the pick product data 112 is updated to "completed."

[0128] Next, an example of data stored in the storage unit 110 of the management device 100 shown in Fig. 15 will be described. Fig. 16 and Fig. 17 are diagrams showing examples of the purchase candidate product data 111 and the pick product data 112 shown in Fig. 15.

[0129] 16 associates the following states with the basket ID "123": the face image is "012301.jpg", the related face image is "012302.jpg", and the price is "2050" yen. Furthermore, the following states are associated as product information for the basket ID "123": the product name is "tomato", the quantity is "3", the cashier status is "yet to be", the product name is "cabbage", the quantity is "1", the cashier status is "yet to be", the product name is "beef", the quantity is "2", and the cashier status is "yet to be".

[0130] The purchase intention product data 111 also associates the face image "012401.jpg" and the price "1120" yen with the basket ID "456." Furthermore, the product information associated with the basket ID "456" is the product name "sauce," the quantity "1," the cashier status "yet to be," the product name "pepper," the quantity "3," the cashier status "yet to be," the product name "banana," the quantity "1," and the cashier status "yet to be."

[0131] 17 associates a state in which the basket ID is "123" with the facial image "012301.jpg." Additionally, as product information corresponding to the facial image "012301.jpg," the product name "tomato" is associated with a state in which the quantity is "3," the removal is "10:15," and the basket management status is "completed," the product name "cabbage" is associated with a state in which the quantity is "1," the removal is "10:17," the return is "10:18," and the basket management status is "returned," and the product name "beef" is associated with a state in which the quantity is "2," the removal is "10:19," and the basket management status is "completed."

[0132] The picked product data 112 associates the facial image "012302.jpg" with a basket ID of "123." The product information corresponding to the facial image "012302.jpg" is also associated with the product name "gum," with the quantity being "1," the time of removal being "10:20," and the basket management status being "not yet."

[0133] Furthermore, the pick product data 112 associates the facial image "012401.jpg" with a state in which the basket ID is "456." Furthermore, as product information corresponding to the facial image "012401.jpg," the product name "sauce" is associated with a state in which the quantity is "1," removal is "10:30," and the basket management status is "completed," the product name "pepper" is associated with a state in which the quantity is "3," removal is "10:31," and the basket management status is "completed," the product name "banana" is associated with a state in which the quantity is "1," removal is "10:32," return is "10:33," the abnormality is "dropped," and the basket management status is "returned," and the product name "banana" is associated with a state in which the quantity is "1," removal is "10:34," and the basket management status is "completed."

[0134] <An example of correction of the purchase intention product data 111 according to the second embodiment> An example of correction of the purchase candidate product data 111 according to the second embodiment will be described below. Fig. 18 is a diagram showing an example of correction of the purchase candidate product data 111 according to the second embodiment.

[0135] As shown in Figure 18(a), if the product names in the purchase intention product data 111 are "tomato," "cabbage," and "beef," and the prize items that the shopper has placed in the shopping basket are unknown, products with a basket management status of "not yet" are identified from the picked product data 112.

[0136] For example, if the picked product data 112 is in the state shown in FIG. 17, the product name is "gum" and the quantity is "1" as a product whose basket management state is "not yet".

[0137] 18(b), the identified products are stored and updated in the purchase candidate product data 111. Specifically, the product names are stored and updated as "tomato," "cabbage," "beef," and "gum."

[0138] <Processing Procedure for Correcting Purchase Intent Product Data 111 According to the Second Embodiment> Next, a description will be given of a processing procedure for correcting the purchase candidate product data 111 according to the present embodiment 2. Fig. 19 is a flowchart showing a processing procedure for correcting the purchase candidate product data 111 according to the present embodiment 2.

[0139] As shown in FIG. 19, when the management device 100 detects that a product has been removed from a product shelf (step S401: Yes), it identifies the removed product, stores the identified product in the pick product data 112, and stores the basket management status as "Not yet" (step S402).

[0140] If the action of putting an item into the shopping basket or shopping cart has not been detected (step S403: No), the process proceeds to step S401.

[0141] If an action of putting an item into a shopping basket or shopping cart is detected (step S403: Yes), it is determined whether the item put into the basket has been identified (step S404).

[0142] If the product placed in the basket is the product taken out in step S401 and the product has already been identified (step S404: Yes), the basket management status of the picked product data 112 for this product is updated to "Completed" (step S405). Then, this product is stored in the purchase candidate product data 111 (step S406), and the process ends.

[0143] If the product placed in the basket is not the product taken out in step S401 and the product cannot be identified (step S404: No), the product whose basket management status is "Not yet" is identified from the pick product data 112 (step S407), and the identified product is stored in the purchase candidate product data 111 (step S408). The basket management status of the pick product data 112 related to the identified product is updated to "Completed" (step S409), and the process ends.

[0144] As described above, the fraud detection system of this embodiment 2 is configured to associate multiple shoppers who visit the store with a single basket ID, identify the items picked by each shopper as a single item they intend to purchase, determine whether they match the items being checked out at the self-checkout, and alert the store clerk if there is a mismatch, thereby efficiently detecting fraudulent behavior when checking out products at the self-checkout.

[0145] Note that the configurations illustrated in the above embodiments are merely functional schematics and do not necessarily have to be physically configured as shown. In other words, the distribution and integration of each device is not limited to that illustrated, and all or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. [Industrial Applicability]

[0146] The fraud detection device, fraud detection system, fraud detection method, and fraud detection program according to the present invention are suitable for efficiently detecting fraudulent behavior during product checkout at self-checkout registers. [Explanation of symbols]

[0147] 10, 10a, 10b camera 20. Wireless Router 30 Store clerk terminal 40 Management device 41 Display section 42 Input section 44 Communications Department 45 Storage section 45a Product Management Data 45b Shelf Data 45c Facial image data 45d Purchase plan product data 45e Checkout product data 45f Report history data 46 Control Unit 46a Product Management Department 46b Product shelf management department 46c Video acquisition unit 46d Facial image acquisition unit 46e Face image determination unit 46f Basket ID acquisition part 46g Human motion detection unit 46h Product identification section 46i Cashier ID Identification Unit 46j Registered Product Identification Section 46k Product Judgment Department 46l Notification Department 46m Data deletion section 50 Self-checkout 81 CPU 82 ROM 83 RAM 84 Non-volatile memory 85 Bus 100 Management device 110 Storage section 111 Purchase plan product data 112 Pick Product Data 120 control section 121 Basket ID acquisition part 122 Pick Product Identification Department 123 Purchasing Product Identification Section

Claims

1. A fraud detection device for a self-checkout system in which a customer performs a scanning operation to register a product, an image acquisition unit that acquires a product pick image that is an image of the product to be picked up from a container installed in a store and that is used when picking up products, and that is an image that is recognizable by image recognition of identification information that is assigned to the container and the product to be picked up from the container; a storage unit that stores product information of the product identified by the product pick image and identification information of the container in association with each other; a registered product identification unit that identifies product information of products registered in the self-service checkout system; a product confirmation unit that confirms whether product information of the product identified by the registered product identification unit is consistent with product information of the product stored in association with the identification information of the container in which the product was stored; a notification unit that issues an alert when the product confirmation unit confirms that the products are not consistent; A fraud detection device comprising:

2. The container is 2. The fraud detection device according to claim 1, wherein the fraud detection device is a shopping basket or a shopping cart.

3. The identification information of the container is 2. The fraud detection device according to claim 1, wherein the information is any one of number information, pattern information, and code information that uniquely identifies the container.

4. The registered product identification unit The fraud detection device according to claim 1 or 3, characterized in that product information of registered products is acquired from the self-service checkout system via an interface.

5. The image acquisition unit A screen image when the product is registered in the self-checkout system is also acquired, The registered product identification unit Identifying product information of the product by performing character recognition on the screen image 4. The fraud detection device according to claim 1 or 3.

6. The product confirmation unit If there is no consistency, extract the inconsistent part, The notification unit The extracted result extracted by the product confirmation unit is notified.

2. The fraud detection device according to claim 1.

7. The product confirmation unit If any of the product information of the products stored in association with the identification information of the container is not included in the product information of the products identified by the registered product identification unit, it is confirmed that the product has not been registered; The notification unit Notify that the product has been omitted from registration 2. The fraud detection device according to claim 1.

8. The product confirmation unit If it is confirmed that the product information of the product identified by the product pick image does not match the product information of the product identified by the registered product identification unit, The notification unit Notify that a product different from the actual product has been registered 2. The fraud detection device according to claim 1.

9. The storage unit 2. The fraud detection device according to claim 1, wherein the stored information is deleted when the product confirmation unit confirms that the product matches.

10. The image acquisition unit A face image of the customer is also acquired together with the product pick image, The storage unit storing a face image of a customer notified by the notification unit in association with the number of notifications; The notification unit Execute a process of displaying a face image of a customer for whom the number of times of notification has exceeded a predetermined threshold.

2. The fraud detection device according to claim 1.

11. A fraud detection system for a self-checkout system in which customers themselves perform a scanning operation to register products, an image acquisition unit that acquires a product pick image that is an image of a recognizable ID information assigned to a container installed in a store and used when picking products, and a recognizable product pick image of the product to be picked from the container; a storage unit that stores product information of the product identified by the product pick image and identification information of the container in association with each other; a registered product identification unit that identifies product information of products registered in the self-service checkout system; a product confirmation unit that confirms whether product information of the product identified by the registered product identification unit is consistent with product information of the product stored in association with the identification information of the container in which the product was stored; a notification unit that issues an alert when the product confirmation unit confirms that the products are not consistent; A fraud detection system comprising:

12. A fraud detection method in a self-checkout system in which a customer performs a scanning operation to register a product, comprising: an image acquisition step of acquiring a product pick image in which image-recognizable identification information assigned to a container installed in a store and used when picking products and the product to be picked from the container can be recognized; a storage step of storing product information of the product identified by the product pick image and identification information of the container in association with each other; a registered product identification step of identifying product information of products registered in the self-checkout system; a product confirmation step of confirming whether or not product information of the product identified in the registered product identification step matches product information of the product stored in association with the identification information of the container in which the product was contained; a notification step of issuing an alert when a mismatch is confirmed in the product confirmation step; 10. A fraud detection method comprising:

13. A fraud detection program executed by a fraud detection device in a self-service checkout system in which customers themselves perform a scanning operation to register products, an image acquisition step of acquiring a product pick image in which image-recognizable identification information assigned to a container installed in a store and used when picking products and the product to be picked from the container can be recognized; a storage step of storing product information of the product identified by the product pick image and identification information of the container in association with each other; a registered product identification step for identifying product information of a product registered in the self-checkout system; a product confirmation step of confirming whether product information of the product identified in the registered product identification step matches product information of the product stored in association with the identification information of the container in which the product was stored; a notification procedure for issuing an alert when a mismatch is confirmed in the product confirmation procedure; A fraud detection program characterized by causing a computer to execute the above.

Images