Monitoring device, settlement device and program

Abstract

To suppress a fraudulent act of a customer for a settlement device compatible with a full-self system.SOLUTION: A monitoring device includes act recognition means, operation recognition means, fraudulent act detection means, image acquisition means, and output means. The act recognition means recognizes an act of a settling person for a settlement device. The operation recognition means recognizes operation input of the settling person to the settlement device. The fraudulent act detection means detects a fraudulent act of the settling person on the basis of a recognition result by the act recognition means and the operation recognition means. The image acquisition means acquires an image identifying the fraudulent act detected by the fraudulent act detection means from a captured image of a camera that captures the settling person operating the settlement device. The output means outputs the image identifying the fraudulent act.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a monitoring device, a settlement device, and respective programs. [Background technology]

[0002] In recent years, self-service point-of-sale (POS) terminals have been attracting attention in supermarkets and other retail stores from the perspective of reducing labor costs and preventing the spread of infectious diseases. Self-service POS terminals are fully self-service payment devices that allow customers to perform all operations themselves, from registering purchased items to paying. This has led to a demand for the development of technology that can prevent fraudulent behavior by customers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-157420 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the embodiments of the present invention is to provide a technology for preventing customers from committing fraudulent acts on fully self-service payment devices. [Means for solving the problem]

[0005] In one embodiment, the monitoring device includes a behavior recognition means, an operation recognition means, a fraud detection means, an image acquisition means, and an output means. The behavior recognition means recognizes the behavior of a payment person with respect to the payment device. The operation recognition means recognizes operation input by the payment person to the payment device. The fraud detection means detects fraudulent activity by the payment person based on the recognition results by the behavior recognition means and the operation recognition means. The image acquisition means acquires an image identifying the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures the payment person operating the payment device. The output means outputs the image identifying the fraudulent activity. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a fully self-service checkout system. [Figure 2] FIG. 2 is a perspective view showing the external configuration of the self-service POS terminal. [Figure 3] FIG. 3 is a block diagram showing the main circuit configuration of the monitoring device. [Figure 4] FIG. 4 is a schematic diagram showing the data structure of the image memory. [Figure 5] FIG. 5 is a schematic diagram showing the data structure of the state memory. [Figure 6] FIG. 6 is a schematic diagram showing the data structure of the list memory. [Figure 7] FIG. 7 is a schematic diagram showing the data structure of the unauthorized memory. [Figure 8] FIG. 8 is a flowchart for explaining the function of the operation recognition unit. [Figure 9] FIG. 9 is a flowchart for explaining the function of the behavior recognition unit. [Figure 10] FIG. 10 is a flowchart for explaining the functions of the fraud detection unit, image acquisition unit, and output unit. [Figure 11] FIG. 11 is a flowchart for explaining the functions of the fraud detection unit, image acquisition unit, and output unit. [Figure 12] FIG. 12 is a flowchart for explaining the functions of the fraud detection unit, image acquisition unit, and output unit. [Figure 13] FIG. 13 is a flowchart for explaining the functions of the fraud detection unit, image acquisition unit, and output unit. [Figure 14] FIG. 14 is a flowchart for explaining the functions of the fraud detection unit, image acquisition unit, and output unit. [Figure 15] FIG. 15 is a schematic diagram showing an example of the product registration screen. [Figure 16] FIG. 16 is a schematic diagram showing an example of a screen on the attendant terminal. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, the embodiments will be described with reference to the drawings.

[0008] [Checkout system explanation] 1 is a schematic diagram showing a configuration of a full self-service checkout system 100. The checkout system 100 includes a plurality of self-service POS terminals 10, an attendant terminal 20, and a store server 30. The checkout system 100 connects each of the self-service POS terminals 10, the attendant terminal 20, and the store server 30 via a communication network 40 such as a LAN (Local Area Network). The store server 30 provides various services related to the checkout system 100 to each of the self-service POS terminals 10 and the attendant terminals 20 connected via the communication network 40.

[0009] The multiple self-service POS terminals 10 are installed at the checkout area of ​​the store, and are operated by customers who have finished shopping. A customer places purchased items in a shopping basket or the like in a sales area where items are displayed and moves to the checkout area. The customer operates the self-service POS terminal 10 to self-pay for the transaction, from registering the purchased items to paying for them. The self-service POS terminal 10 is an example of a payment device. The payment device can also be referred to as a payment terminal, a checkout device, an accounting terminal, a product registration device, a product registration terminal, etc.

[0010] The attendant terminal 20 is installed in a monitoring station located near the exit of the checkout area. The attendant terminal 20 displays, on a display device, a list of purchased items registered for each self-service POS terminal 10. If an error occurs in the self-service POS terminal 10, the attendant terminal 20 notifies the customer of the error. The store clerk at the monitoring station, i.e., the attendant, monitors the checkout area based on the information displayed and notified on the attendant terminal 20. Although FIG. 1 shows one attendant terminal 20, multiple attendant terminals 20 may be connected to the communication network 40.

[0011] Each self-service POS terminal 10 includes a monitoring device 50. The monitoring device 50 may be located either inside or outside the self-service POS terminal 10. The monitoring device 50 monitors the customer operating the self-service POS terminal 10, i.e., the person making the payment. Specifically, the monitoring device 50 monitors the person operating the self-service POS terminal 10 to pay for a transaction based on information such as images captured by a camera 60 installed near the self-service POS terminal 10, and issues an alert if the person engages in fraudulent behavior. Fraudulent behavior includes, for example, bagging purchased items without registering them in the self-service POS terminal 10. Fraudulent behavior includes, for example, leaving the checkout area without paying for the purchased items. The alert is sent to both the self-service POS terminal 10 and / or the attendant terminal 20. Each monitoring device 50 is connected to the attendant terminal 20 via the communication network 40 and can output an alert signal to the attendant terminal 20. Each monitoring device 50 may be connected to the attendant terminal 20 without going through the communication network 40 .

[0012] [Self-POS terminal explanation] 2 is a perspective view showing the external configuration of the self-service POS terminal 10. The self-service POS terminal 10 includes a main body 101, and a first product placement table 102 and a second product placement table 103, which are arranged on the left and right sides of the main body 101. The first product placement table 102, which is arranged on the right side as viewed from the main body 101, is a place where a customer who is making a payment can place purchased products before registration. Purchased products before registration are usually placed in a shopping basket and placed on the first product placement table 102. Purchased products before registration may also be placed in a shopping cart and placed above the first product placement table 102.

[0013] The second product placement table 103, located on the left side of the main body 101, is a place where the purchaser places the purchased items after registering them. After registration, the purchased items are placed in a container placed on the second product placement table 103, such as a shopping basket, a plastic shopping bag, or a personal shopping bag. After registration, the purchased items may be placed directly on the second product placement table 103. In FIG. 2 , the second product placement table 103 has a temporary placement table 104 attached to its upper part via a support pillar. The temporary placement table 104 is, for example, a table on which the purchaser can temporarily place the items before placing them in a plastic shopping bag or the like. The temporary placement table 104 has two holding arms 105 spaced apart on the left and right. Using these holding arms 105, the purchaser can place an open plastic shopping bag, personal shopping bag, or the like on the second product placement table 103.

[0014] The main body 101 is equipped with a touch panel 11, a card reader 12, a printer 13, a scanner 14, a hand scanner 15, a cash processing machine 16, a speaker 17, and the like as input / output devices.

[0015] The touch panel 11 includes a display as a display device and a touch sensor as an input device. The display displays various screen information such as a payment start screen, a product registration screen, a payment method selection screen, and an accounting screen. The payment start screen is a screen that accepts the start of a transaction payment. A start key is provided on the payment start screen to command the start of a transaction payment. A payer who begins registering purchased products touches the start key. When the touch sensor detects that the start key has been touched, the controller of the self-service POS terminal 10 switches the display screen from the payment start screen to the product registration screen.

[0016] The product registration screen displays detailed information such as the product name, price, and total amount of the registered purchased products. A checkout key is provided on the product registration screen to instruct the user to proceed to payment for the registered purchased products. After registering the purchased products, the user touches the payment key. When the touch sensor detects that the checkout key has been touched, the controller of the self-service POS terminal 10 switches the display screen from the product registration screen to a payment method selection screen.

[0017] The payment method selection screen is a screen that accepts the selection input of a payment method such as cash payment, credit card payment, or electronic money payment. The payment method selection screen is arranged with a cash key for issuing a cash payment command, a credit key for issuing a credit card payment command, an electronic money key for issuing an electronic money payment command, and the like. For example, a person wishing to pay in cash touches the cash key. When the touch sensor detects that the cash key has been touched, the controller of the self-service POS terminal 10 switches the display screen from the payment method selection screen to the checkout screen. The checkout screen is a screen that displays the total amount of the transaction, the amount of cash to be deposited, the amount of change, etc.

[0018] The card reader 12 is a device that reads data recorded on a card medium such as a credit card, electronic money card, or point card. If the card medium is a magnetic card, the card reader 12 is a magnetic card reader. If the card medium is an IC card, the card reader 12 is an IC card reader. The self-service POS terminal 10 may be equipped with either a magnetic card reader or an IC card reader as the card reader 12, or may be equipped with both. The card reader 12 may also be a card reader / writer that has a function for writing data to a card.

[0019] The printer 13 is a device that prints data related to purchase receipts, credit card slips, etc. onto receipt paper. The receipt paper on which various data has been printed by the printer 13 is cut by a cutter and issued from the receipt issuing slot. The printer 13 may be implemented as, for example, a thermal printer or a dot-matrix printer.

[0020] The scanner 14 and the hand scanner 15 are used to read information identifying a product. In this embodiment, the scanner 14 and the hand scanner 15 read barcodes attached to products as information identifying the products. The scanner 14 optically reads the barcode of a product held up against a reading window (glass window) so that the barcode on the product faces the scanner 14. The hand scanner 15 is handheld and operated by the person making the payment, and optically reads the barcode when brought close to the barcode on the product. The scanner 14 and the hand scanner 15 may be used to read information identifying a product, and may also be used to read information other than a barcode, such as a two-dimensional code.

[0021] The cash processing machine 16 processes cash. The cash processing machine 16 has a banknote input port 161, a banknote discharge port 162, a coin input port 163, and a coin discharge port 164. The cash processing machine 16 processes banknotes inserted into the banknote input port 161. The cash processing machine 16 discharges banknotes as change from the banknote discharge port 162. The cash processing machine 16 accepts and processes coins inserted into the coin input port 163. The cash processing machine 16 discharges coins as change from the coin discharge port 164.

[0022] The speaker 17 outputs a warning sound as an alert or a voice message.

[0023] A camera 60 serving as an imaging device is attached to the upper part of the main body 101. The camera 60 photographs a person operating the self-service POS terminal 10. As shown in FIG. 2 , the display of the touch panel 11, the card insertion slot of the card reader 12, the receipt issuing slot of the printer 13, the reading window of the scanner 14, and the bill insertion slot 161, the bill ejection slot 162, the coin insertion slot 163, and the coin ejection slot 164 of the cash processing machine 16 are arranged on one surface (hereinafter referred to as the front) of the main body 101. Therefore, the person operating the self-service POS terminal 10 stands facing the front of the main body 101. The camera 60 photographs almost the entire body of the person operating the self-service POS terminal 10 from above. Therefore, the head, face, both shoulders, both arms, chest, abdomen, etc. of the person operating the self-service POS terminal 10 are photographed by the camera 60.

[0024] [Configuration of monitoring device] 3 is a block diagram showing the main circuit configuration of the monitoring device 50. The monitoring device 50 includes a processor 51, a main memory 52, an auxiliary storage device 53, a clock 54, a camera interface 55, a terminal interface 56, a network interface 57, and a system transmission path 58. The system transmission path 58 includes an address bus, a data bus, a control signal line, etc. The system transmission path 58 connects the processor 51 to each of the other components and transmits data signals exchanged between them.

[0025] The monitoring device 50 is configured as a computer by connecting a processor 51, a main memory 52, an auxiliary storage device 53, a clock 54, a camera interface 55, a terminal interface 56, and a network interface 57 via a system transmission line 58.

[0026] The processor 51 corresponds to the central part of the computer. The processor 51 controls each part to realize various functions of the monitoring device 50 in accordance with an operating system or an application program. The processor 51 is, for example, a CPU (Central Processing Unit).

[0027] The main memory 52 corresponds to the main storage portion of the computer. The main memory 52 includes a nonvolatile memory area and a volatile memory area. The main memory 52 stores an operating system or application programs in the nonvolatile memory area. The main memory 52 may store data required for the processor 51 to execute processes for controlling each part in either the nonvolatile or volatile memory area. The main memory 52 uses the volatile memory area as a work area where data is rewritten by the processor 51 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0028] The auxiliary storage device 53 corresponds to the auxiliary storage portion of the computer. For example, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 53. The auxiliary storage device 53 stores data used by the processor 51 when performing various processes, data created by the processes in the processor 51, etc. The auxiliary storage device 53 may also store the above-mentioned application programs.

[0029] The camera interface 55 connects to the camera 60 provided in the self-service POS terminal 10. The camera interface 55 outputs a control signal to the camera 60 to control the operation of the camera 60. The camera 60 performs a photographing operation in accordance with this control signal. The camera interface 55 captures image data photographed by the camera 60.

[0030] The terminal interface 56 connects to the self-service POS terminal 10. The terminal interface 56 performs data communication with the self-service POS terminal 10.

[0031] The network interface 57 connects to the attendant terminal 20 via the communication network 40. The network interface 57 performs data communication with the attendant terminal 20.

[0032] The monitoring device 50 configured as described above uses part of the volatile memory area in the main memory 52 as an image memory 521 , a status memory 522 , a list memory 523 , and an invalid memory 524 .

[0033] 4 is a schematic diagram showing the data structure of image memory 521. Image memory 521 is an area for storing a plurality of frame images captured by camera 60 at a given time, in association with the time of capture.

[0034] 5 is a schematic diagram showing the data structure of the status memory 522. The status memory 522 is an area for storing multiple status data in association with transition times. The status data is data indicating various states related to the actions of the person operating the self-service POS terminal 10 and the operation input by the person to the self-service POS terminal 10. Details of the states will be described later. The transition time is the time when a transition occurs to the state indicated by the status data.

[0035] 6 is a schematic diagram showing the data structure of the list memory 523. The list memory 523 is an area for storing multiple pieces of registered product data in association with the time of registration. The registered product data is data related to products registered at the self-service POS terminal 10, and includes product names and product images. The registered product data may further include data such as price and number of items purchased. The registered product data may omit product images.

[0036] FIG. 7 is a schematic diagram showing the data structure of the fraud memory 524. The fraud memory 524 is an area for storing multiple pieces of fraud data in association with the recognition time. The fraud data includes a fraud classification, a count, and a fraud recognition image. The fraud classification is information that identifies the type of fraud. The types of fraud will be described later. The count is the number of times fraud has occurred. The fraud recognition image is an image that can identify fraud.

[0037] Returning to the explanation of Figure 3. The processor 51 has at least the functions of an operation recognition unit 511, a behavior recognition unit 512, a fraud detection unit 513, an image acquisition unit 514, and an output unit 515. The operation recognition unit 511 has a function of recognizing an operation input by the settler to the self-service POS terminal 10. The operation recognition unit 511 can be referred to as an operation recognition means. The behavior recognition unit 512 has a function of recognizing the settler's behavior with respect to the self-service POS terminal 10. The behavior recognition unit 512 can be referred to as a behavior recognition means. The fraud detection unit 513 has a function of detecting fraudulent behavior by the settler based on the recognition results by the behavior recognition unit 512 and the operation recognition unit 511. The fraud detection unit 513 can be referred to as a fraud detection means. The image acquisition unit 514 has a function of acquiring an image identifying fraudulent behavior detected by the fraud detection unit 513 from an image captured by the camera 60 that captures an image of the settler operating the self-service POS terminal 10. The image acquisition unit 514 can be referred to as an image acquisition means. The output unit 515 has a function of outputting an image that identifies fraudulent activity to the self-POS terminal 10 or the attendant terminal 20. The output unit 515 can be referred to as an output means.

[0038] The functions of the operation recognition unit 511, the behavior recognition unit 512, the fraud detection unit 513, the image acquisition unit 514, and the output unit 515 are realized by information processing that the processor 51 executes in accordance with the payer monitoring program.

[0039] The settlor monitoring program is a type of application program stored in the main memory 52 or the auxiliary storage device 53. There are no particular limitations on the method for installing the settlor monitoring program in the main memory 52 or the auxiliary storage device 53. The settlor monitoring program can be recorded on a removable recording medium, or can be distributed via communication over a network and installed in the main memory 52 or the auxiliary storage device 53. The recording medium can be in any form, such as a CD-ROM or memory card, as long as it can store the program and is readable by the device.

[0040] [Functional description of monitoring device] Next, the functions of the processor 51 of the monitoring device 50 as the operation recognition unit 511, behavior recognition unit 512, fraud detection unit 513, image acquisition unit 514, and output unit 515 will be described using the flowcharts in FIGS. 8 to 14 and the screen examples in FIGS. 15 and 16. Note that the procedures and contents of the flowcharts described below are examples. The procedures and contents can be changed as appropriate as long as similar effects can be achieved. The screen examples are also examples. The layout, display items, etc. can be changed as appropriate as long as similar effects can be achieved.

[0041] 8 is a flow chart for explaining the function of the operation recognition unit 511. After finishing their shopping, the payer enters the checkout area and stands in front of an empty self-service POS terminal 10. The payer then places the shopping basket containing the purchased items that have not yet been registered on the first product placement table 102, and places the empty shopping basket, shopping bags, reusable bags, and other containers on the second product placement table 103. A payment start screen is displayed on the touch panel 11 of the self-service POS terminal 10. The payer touches the start key arranged on the payment start screen.

[0042] The processor 51 of the monitoring device 50 waits for operation input data for starting use as ACT1 in FIG. 8. The operation input data for starting use is output from the self-service POS terminal 10 to the monitoring device 50 in response to touching the start key on the payment start screen. When the processor 51 inputs the operation input data for starting use, the process proceeds to ACT2. As ACT2, the processor 51 initializes the image memory 521, the status memory 522, the list memory 523, and the fraud memory 524. Furthermore, as ACT3, the processor 51 acquires the time measured by the clock 54 as a transition time, and stores a status "11" in the status memory 522 in association with the transition time. In other words, the status "11" indicates a state in which the operation input by the settler to the self-service POS terminal 10 for starting use has been recognized.

[0043] The processor 51 notifies the start of behavior recognition as ACT4. This notification is sent to the processor 51 itself. Upon receiving this notification, the processor 51 starts functioning as the behavior recognition unit 512. That is, the processor 51 executes the function as the behavior recognition unit 512 in parallel with the function as the operation recognition unit 511. Here, for convenience of explanation, the function as the behavior recognition unit 512 will be explained first.

[0044] FIG. 9 is a flowchart illustrating the function of the behavior recognition unit 512. When the processor 51 starts functioning as the behavior recognition unit 512, the processor 51 outputs an activation signal to the camera 60 in ACT21. The camera 60 starts capturing images upon receiving the activation signal. The processor 51 captures images captured by the camera 60 in ACT22. The captured images may be still images or videos. The processor 51 recognizes the behavior of the person standing in front of the self-service POS terminal 10 based on the captured images in ACT23. Specifically, the processor 51 estimates the skeletal structure of the person from the captured images. The skeletal structure estimation is performed using AI technology such as deep learning. The processor 51 recognizes the behavior of the person taking out, registering, bagging, or leaving from, for example, hand movements obtained by skeletal structure estimation. The function of the behavior recognition unit 512 may also be realized by motion detection or the like that detects the operator's movements.

[0045] A take-out action is an action in which the payer takes out an item from a shopping basket placed on the first item placement table 102. For example, an action in which the payer takes out an item from the first item placement table 102 before the item is registered and puts it in a customer's pocket or the like without putting it in a bag is also a take-out action. A registration action is an action in which the payer holds an item over the reading window of the scanner 14 to have the scanner 14 read the barcode attached to the item. A bagging action is an action in which the payer places an item in a container prepared on the second item placement table 103. Note that an action in which the payer places an item on the second item placement table 103 or the temporary placement table 104 without putting it in a container is also a type of bagging action. An exit action is an action in which the payer moves away from the self-service POS terminal 10 to exit the checkout area.

[0046] The processor 51 checks whether a take-out action has been recognized in ACT 24. If a take-out action has not been recognized, the processor 51 checks whether a registered action has been recognized in ACT 25. If a registered action has not been recognized, the processor 51 checks whether a bagging action has been recognized in ACT 26. If a bagging action has not been recognized, the processor 51 checks whether an exit action has been recognized in ACT 27. If an exit action has not been recognized, the processor 51 checks whether to end the action recognition in ACT 28. As will be described later, the end of the action recognition is notified during the function as the operation recognition unit 511. If the action recognition is not to be ended, the processor 51 returns to ACT 22.

[0047] In this way, processor 51 sequentially captures images (video) captured by camera 60 until it is notified that behavior recognition has ended, and recognizes the behavior of the payer based on the captured images. As a result, if processor 51 recognizes a take-out behavior, processor 51 proceeds to ACT 29. Processor 51 stores multiple frame images in which a take-out behavior has been recognized in ACT 29 in image memory 521 along with the time of capture. An image in which a take-out behavior has been recognized is an image that shows the hand movements of the payer when taking an item from the shopping cart. Processor 51 acquires the time measured by clock 54 as the transition time in ACT 30, and stores status "21" in status memory 522 in association with that transition time. In other words, status "21" indicates a state in which the take-out behavior of the payer has been recognized.

[0048] After completing the processing of ACT29 and ACT30, the processor 51 returns to ACT22. The processor 51 again sequentially captures images (video) captured by the camera 60 and recognizes the behavior of the payer based on the captured images. As a result, if the processor 51 recognizes the registered behavior, the processor 51 proceeds to ACT31. The processor 51 stores multiple frame images from which the registered behavior has been recognized in ACT31 in the image memory 521 along with the time of capture. The images from which the registered behavior has been recognized are images that show the hand movements of the payer as they hold the item over the reading window of the scanner 14. The processor 51 acquires the time measured by the clock 54 as the transition time in ACT32, and stores the status "22" in the status memory 522 in association with the transition time. In other words, the status "22" indicates the state in which the registered behavior of the payer has been recognized.

[0049] After completing the processes of ACT31 and ACT32, processor 51 returns to ACT22. Processor 51 again sequentially captures images (video) captured by camera 60 and recognizes the actions of the payer based on the captured images. If the processor 51 recognizes the bagging action as a result, it proceeds to ACT33. In ACT33, processor 51 stores multiple frame images in image memory 521 from which the bagging action has been recognized along with the time of capture. An image from which the bagging action has been recognized is an image that shows the hand movements of the payer as they place the products in the container. Alternatively, it is an image that shows the hand movements of the payer as they place the products on the second product placement table 103 or the temporary placement table 104. In ACT34, processor 51 acquires the time measured by clock 54 as the transition time, and stores status "23" in status memory 522 in association with the transition time. In other words, status "23" indicates a state in which the processor 51 has recognized the bagging action of the payer.

[0050] After completing the processing of ACT33 and ACT34, the processor 51 returns to ACT22. The processor 51 again sequentially captures images (video) captured by the camera 60 and recognizes the behavior of the settler based on the captured images. As a result, if the processor 51 recognizes an exit behavior, the processor 51 proceeds to ACT35. The processor 51 stores multiple frame images in which the exit behavior has been recognized in ACT35 in the image memory 521 along with the time of capture. An image in which the exit behavior has been recognized is an image that shows the movement of the settler moving away from the self-service POS terminal 10. The processor 51 acquires the time measured by the clock 54 as the transition time in ACT36, and stores the status "24" in the status memory 522 in association with the transition time. In other words, the status "24" indicates a state in which the exit behavior of the settler has been recognized.

[0051] When the processor 51 completes the processing of ACT 35 and ACT 36, it returns to ACT 22. The processor 51 again sequentially captures images (video) captured by the camera 60 and recognizes the behavior of the payer based on the captured images. Then, when the processor 51 recognizes the taking action, the registration action, the bagging action, or the exit action, it executes the processing of ACT 29 to ACT 36, respectively, in the same manner as described above.

[0052] When the processor 51 is notified of the end of behavior recognition, the processor 51 proceeds to ACT 37. In ACT 37, the processor 51 outputs a stop signal to the camera 60. The camera 60 receives the stop signal and stops capturing images. With this, the processor 51 ends its function as the behavior recognition unit 512.

[0053] Returning to the functional description of the operation recognition unit 511. When the person making the payment touches the start key on the payment start screen, the screen on the touch panel 11 switches from the payment start screen to the product registration screen SCa (see FIG. 15).

[0054] 15 is an example of a product registration screen SCa. As shown in the figure, the product registration screen SCa includes a message display area Ma, a list display area La, a total display area Aa, a registered product image display area Pa, a fraud specific image display area Qa, a checkout key Bx, a cancel key By, and a store clerk call key Bz.

[0055] The message display area Ma displays a message indicating how to register a product, such as "Please hold the product barcode close to the reading window." The list display area La displays the product name, number of items, price, etc. of registered products in list format. The total display area Aa displays the total number of items and total price of registered products. The registered product image display area Pa displays images of products registered as purchased products. The fraud identification image display area Qa displays images and warning messages for identifying fraudulent activity acquired using the functions of the image acquisition unit 514. The checkout key Bx is a soft key used by the payer who has finished registering the purchased products to start checkout. The cancel key By is a soft key used by the payer to cancel the registration of the purchased products. The clerk call key Bz is a soft key used by the payer to call a clerk.

[0056] Fig. 16 is an example of a screen SCb displayed on the display device of the attendant terminal 20 for the self-service POS terminal 10 on which the product registration screen SCa of Fig. 15 is displayed. On the display device of the attendant terminal 20, screens SCb corresponding to the product registration screens SCa of the respective self-service POS terminals 10 are separated, and a plurality of screens SCb are displayed.

[0057] The screen SCb includes a list display area Lb, a total display area Ab, and a fraudulent specific image display area Qb. The list display area Lb displays the same content as the list display area La of the product registration screen SCa. The total display area Ab displays the same content as the total display area Aa of the product registration screen SCa. The fraudulent specific image display area Qb displays the same content as the fraudulent specific image display area Qa of the product registration screen SCa. However, the warning message displayed in the fraudulent specific image display area Qa is intended for the person making the payment, i.e., the customer, whereas the warning message displayed in the fraudulent specific image display area Qb is intended for the attendant, i.e., the store clerk.

[0058] The processor 51, which has notified the start of behavior recognition in ACT4 of FIG. 8, waits for operation input data from the self-service POS terminal 10 in ACT5. The operation input data includes operation input data related to product registration, operation input data related to an instruction to cancel registration, operation input data related to an instruction to start checkout, etc. The operation input data related to product registration is product data identified by the barcode scanned by the scanner 14 when the payer holds the product's barcode over the reading window. The product data includes at least the product name. The operation input data related to an instruction to cancel registration is data of a stop key signal output from the self-service POS terminal 10 when the payer touches the stop key By on the product registration screen SCa. The operation input data related to an instruction to start checkout is data of a checkout key signal output from the self-service POS terminal 10 when the payer touches the checkout key Bx on the product registration screen SCa.

[0059] When processor 51 receives operation input data from self-service POS terminal 10, it checks in ACT6 whether the operation input data is related to product registration, operation input data related to an instruction to cancel registration, or operation input data related to an instruction to start checkout. If any other operation input data is received, processor 51 returns to ACT5 and waits for the next operation input data.

[0060] If operation input data related to product registration is input in the operation input data waiting state of ACT5 to ACT8, the processor 51 proceeds to ACT9. The processor 51 acquires the time measured by the clock 54 in ACT9 as the registration time, and stores the status "12" in the status memory 522 in association with the registration time. In other words, the status "12" indicates a state in which operation input data related to product registration has been recognized.

[0061] In ACT10, processor 51 acquires a registered product image. For example, processor 51 cuts out a product image from a plurality of frame images that have been stored in image memory 521 just before the registration time and that have been recognized as registered actions, and acquires the image as the registered product image. The product image is an image of an object that the payer is holding over the reading window. The product image may include part of the payer's hand. If the operation input data related to product registration includes data of a preset product image, the registered product image may be the product image included in that operation input data. In ACT11, processor 51 stores the product name and registered product image included in the operation input data related to product registration in list memory 523, in association with the registration time. Thereafter, processor 51 returns to ACT5 and waits for the next operation input data.

[0062] In the operation input data waiting state of ACT5 to ACT8, if operation input data related to registration cancellation is input in ACT7, the processor 51 proceeds to ACT12. The processor 51 acquires the time measured by the clock 54 in ACT12 as the registration cancellation time, and stores the status "15" in the status memory 522 in association with the registration cancellation time. In other words, the status "15" indicates a state in which operation input data related to registration cancellation has been recognized.

[0063] The processor 51 notifies the end of behavior recognition in ACT13. This notification is sent to the processor 51 itself. Upon receiving this notification, the processor 51 ends its function as the behavior recognition unit 512. Furthermore, the processor 51 saves the data in the state memory 522 in the auxiliary storage device 53 in ACT14. With this, the processor 51 ends its function as the operation recognition unit 511.

[0064] When operation input data related to the start of a transaction is entered in ACT8 while in the operation input data standby state from ACT5 to ACT8, processor 51 proceeds to ACT15. In ACT15, processor 51 obtains the time measured by clock 54 as the transaction start time, and stores status "13" in status memory 522 in association with that transaction start time. In other words, status "13" indicates a state in which operation input data related to the start of a transaction has been recognized.

[0065] In ACT 16, processor 51 waits for transaction completion data. The transaction completion data is data that is output from self-service POS terminal 10 when the self-service POS terminal 10 accepts the selection input of a payment method, such as cash payment, credit card payment, or electronic money payment, and payment is completed using the selected payment method. When the transaction completion data is input, processor 51 proceeds to ACT 17. In ACT 17, processor 51 obtains the time measured by clock 54 as the transaction completion time, and stores status "14" in status memory 522 in association with that transaction completion time. In other words, status "14" indicates a state in which operational input data related to the transaction completion has been recognized.

[0066] Thereafter, the processor 51 proceeds to ACT13 described above. That is, the processor 51 notifies the end of behavior recognition. The processor 51 also saves the data in the state memory 522 in the auxiliary storage device 53. With this, the processor 51 ends its function as the operation recognition unit 511.

[0067] The processor 51 executes processing as the operation recognition unit 511 shown in ACT1 to ACT17 of Fig. 8 every time operation input data for starting use is input from the self-service POS terminal 10. The processor 51 also executes processing as the behavior recognition unit 512 shown in ACT21 to ACT37 of Fig. 9. Thus, the auxiliary storage device 53 stores data in a status memory 522 that stores various statuses related to each person who has made a self-payment using the self-service POS terminal 10 in chronological order. Therefore, by analyzing the data in the status memory 522, it is possible to verify the operation history and behavior history of the person who has made the payment.

[0068] On the other hand, the monitoring device 50 detects fraudulent acts by the person making the payment in real time, warns the person making the payment, and notifies the attendant, thereby preventing fraud before it occurs. Such fraud prevention function is realized by the fraud detection unit 513, the image acquisition unit 514, and the output unit 515.

[0069] 10 to 14 are flow charts for explaining the functions of the fraud detection unit 513, the image acquisition unit 514, and the output unit 515. In Fig. 10, the processor 51 waits for the status "11" to be stored in the state memory 522 as ACT41.

[0070] When the status "11" is stored in the status memory 522, the processor 51 proceeds to ACT42. The processor 51 acquires the next status data from the status memory 522 in ACT42. The processor 51 then waits to acquire the status "21" in ACT43 or the status "24" in ACT44. That is, when the processor 51 recognizes an operation input by the settler to start use of the self-checkout POS terminal 10, the processor 51 waits to recognize the settler's removal action or the settler's exit action.

[0071] If the processor 51 acquires the status "21" in the standby state of ACT43 and ACT44, it proceeds to ACT45. The processor 51 acquires the next status data from the status memory 522 in ACT45. The processor 51 then waits to acquire the status "22" in ACT46, the status "23" in ACT47, or the status "24" in ACT48. In other words, when the processor 51 recognizes the take-out action of the payer, it waits to recognize the registration action, the bagging action, or the exit action of the payer.

[0072] In the standby state of ACT46 to ACT48, if the processor 51 acquires the status "22", the processor 51 proceeds to ACT51 in Fig. 11. The processor 51 acquires the next status data from the status memory 522 in ACT51. The processor 51 then waits to acquire the status "23" in ACT52, the status "24" in ACT53, or the status "21" in ACT54. In other words, once the processor 51 recognizes the payment person's registered action, it waits to recognize the payment person's bagging action, exit action, or removal action.

[0073] If the processor 51 acquires the status "23" in the standby state of ACT52 to ACT54, the processor 51 proceeds to ACT55. The processor 51 acquires the status stored immediately before the status "23" from the status memory 522 in ACT55. The processor 51 checks whether the status stored immediately before the status "23" is "12" in ACT56. That is, the processor 51 checks whether the operation input data related to product registration was recognized before the bagging action. If the status "12" is stored, the processor 51 proceeds to ACT57. The processor 51 outputs the latest product registration image stored in the list memory 523 to the self-checkout POS terminal 10 in ACT57. As a result, the self-checkout POS terminal 10 displays the product registration image in the registered product image display area Pa of the product registration screen SCa. This image is an image of the purchased product that was registered by the settlement person by having the scanner 14 of the self-checkout POS terminal 10 read the barcode.

[0074] After completing the process in ACT 57, the processor 51 returns to ACT 42 in Fig. 10. Then, the processor 51 executes the processes from ACT 42 onwards in the same manner as described above.

[0075] In this way, processor 51 recognizes the take-out action of the payer, then recognizes the registration action, recognizes the operation input data related to product registration, and then recognizes the bagging action, and then determines that there is no fraud. Processor 51 outputs an image of the purchased product that has been registered to self-service POS terminal 10.

[0076] Next, a case where fraud by the payer is recognized will be described. First, we will explain the fraud in which the payment person registers a product but then bags the product without registering it. This type of fraud occurs, for example, when the payment person covers the barcode with their hand and holds it over the reading window of scanner 14. In this case, status memory 522 stores status "22," which indicates a registration action, after status "21," which indicates a removal action, and then status "23," which indicates a bagging action. However, status "12," which indicates operational input data related to product registration, is not stored between status "22" and status "23." That is, this occurs when, in ACT 56 of FIG. 11, the status stored immediately before status "23" is not "12" (NO in ACT 56).

[0077] Processor 51 proceeds to ACT61 in FIG. 12. In ACT61, processor 51 acquires an image of the bagging action. Specifically, processor 51 acquires a plurality of frame images that can recognize the bagging action stored in image memory 521 around the time when the status changed to "23". In ACT62, processor 51 extracts an image that can identify the product bagged by the bagging action from the acquired frame images, as an image that identifies fraudulent activity. For example, processor 51 extracts an image of the product being held by the payment person. For example, processor 51 extracts an image of the product that the payment person placed on second product placement table 103 or temporary placement table 104.

[0078] Here, the processor 51 functions as a fraud detection unit 513 by the processing of ACT56 in Fig. 11. The processor 51 also functions as an image acquisition unit 514 by the processing of ACT61 and ACT62 in Fig. 12.

[0079] After completing the processing in ACT62, processor 51 sets the fraud category to "91" in ACT63. The fraud category "91" is information indicating fraud where the product is not registered. Processor 51 counts up the number counter by +1 in ACT64. The number counter was reset to "0" when each memory was initialized in ACT2 of Figure 8.

[0080] The processor 51 stores the fraudulent data in the fraud memory 524 in ACT 65. That is, the processor 51 sets the current time measured by the clock 54 as the fraud recognition time. Then, the processor 51 stores the fraud classification determined in ACT 63, the value of the number counter updated in ACT 64, and the image identifying the fraudulent activity obtained in ACT 62, i.e., the fraud recognition image, in the fraud memory 524, in association with this recognition time.

[0081] The processor 51 outputs the fraudulent identification image and a warning message for the settlement person corresponding to the fraud classification "91" to the self-service POS terminal 10 as ACT66. As a result, the self-service POS terminal 10 displays the fraudulent identification image and the warning message for the settlement person in the fraud identification image display area Qa of the product registration screen SCa. The warning message notifies the settlement person that the product in the image is not registered and that the product needs to be re-registered, for example, by saying "The product is not registered. Please register it again." In this way, for a settlement person who has registered a product but then bagged it without registering it, the fraudulent identification image and the warning message are displayed in the fraud identification image display area Qa.

[0082] The processor 51 checks whether the number of times is equal to or greater than the threshold value N in ACT 67. If the number of times is equal to or greater than the threshold value N, the processor 51 proceeds to ACT 68. In ACT 68, the processor 51 outputs a fraudulent identification image and a warning message for the attendant corresponding to the fraud classification "91" to the attendant terminal 20. As a result, the attendant terminal 20 displays the fraudulent identification image and the warning message for the attendant in the fraud identification image display area Qb on the screen SCb corresponding to the self-service POS terminal 10. The warning message notifies the attendant that a fraudulent product registration has occurred, such as "A product has not been registered." In this way, if the settlor commits fraud N or more times, a report is sent to the attendant via the attendant terminal 20.

[0083] The threshold value N is an arbitrary value equal to or greater than "1." For example, if the attendant terminal 20 is notified of fraudulent activity every time the activity is detected, the threshold value N is set to "1." For example, if the attendant terminal 20 is notified of fraudulent activity when three or more fraudulent activities are detected in a single payment, the threshold value N is set to "3." By setting the threshold value N to "2" or greater, the number of times fraudulent activity is notified to the attendant terminal 20 is reduced, thereby reducing the burden on the attendant. However, there is a risk that the attendant may overlook the fraudulent activity of the payer. If the threshold value N is set to "1," the attendant terminal 20 is notified of fraudulent activity every time the activity is detected, thereby reducing the probability that the attendant will overlook fraudulent activity.

[0084] Here, the processor 51 functions as the output unit 515 by the processing in ACT66 and ACT68.

[0085] If the number of times is less than the threshold N, or if the number of times is equal to or greater than the threshold N and therefore the processor 51 outputs a fraud recognition image and a warning message to the attendant terminal 20, the processor 51 proceeds to ACT 71 in Fig. 13. In ACT 71, the processor 51 acquires the next status data from the status memory 522. Then, the processor 51 waits to acquire status "22" in ACT 72, status "24" in ACT 73, or status "21" in ACT 74. In other words, when the processor 51 detects fraudulent activity of not registering a product, the processor 51 waits to recognize the payment user's registration behavior, exit behavior, or removal behavior.

[0086] When the settlement person confirms the fraud recognition image and warning message displayed in the fraud identification image display area Qa of the product registration screen SCa and wishes to resolve the fraud of not registering the product, he or she removes the product identified in the fraud recognition image from the second product placement table 103 and performs the registration action. That is, this corresponds to the case where the status "22" is acquired (YES in ACT72) in the standby state of ACT72 to ACT74 in Fig. 13.

[0087] The processor 51 proceeds to ACT 75. In ACT 75, the processor 51 cancels the output of the fraud recognition image and the warning message. As a result, the display of the fraud identification image display area Qa is erased in the self POS terminal 10. The display of the fraud identification image display area Qb is erased in the attendant terminal 20.

[0088] After completing the processing of ACT 75, the processor 51 proceeds to ACT 51 in Fig. 11. The processor 51 executes the processing from ACT 51 onwards in the same manner as described above.

[0089] Therefore, when the barcode of a product is read by the scanner 14 as a result of the payment person's registration action, the status memory 522 stores the status "12" after the status "22". Furthermore, as a result of the payment person's subsequent bagging action, the status memory 522 stores the status "23". As a result, the processor 51 sequentially executes the processes of ACT51, ACT55 and ACT57, and returns to ACT42 in Figure 10. In other words, the fraud of the product not being registered is resolved.

[0090] On the other hand, there may be cases where the settlement person repeats the fraudulent act of removing the next product from the first product placement table 102 without correcting the fraudulent unregistered product. This applies to the case where the status "21" is acquired (YES in ACT74) in the standby state of ACT72 to ACT74 in FIG.

[0091] The processor 51 proceeds to ACT 76. In ACT 76, the processor 51 outputs an error command to the self-service POS terminal 10. The self-service POS terminal 10 to which the error command has been input enters an error state and becomes inoperable. The error state will not be released unless an attendant performs an error release operation.

[0092] The processor 51 outputs an error signal to the attendant terminal 20 in ACT77. The error signal includes the fraudulent identification image acquired in ACT62 and a warning message for the attendant corresponding to the fraud category "91." The attendant terminal 20, which has received the error signal, displays the fraudulent identification image and a warning message for the attendant in the fraud identification image display area Qb on the screen SCb corresponding to the self-service POS terminal 10. The attendant terminal 20 also notifies the attendant that an error has occurred in the self-service POS terminal 10. The notification method is not limited. For example, the notification may be by lighting a lamp or sounding an error sound. After confirming the details of the fraud and the error, the attendant instructs the payer to go to the self-service POS terminal 10 and resolve the fraud. The attendant also performs an operation to resolve the error. When the error is resolved, an error resolution signal is output from the self-service POS terminal 10 to the monitoring device 50.

[0093] After outputting the error command and error signal, processor 51 waits for an error cancellation signal in ACT 78. When the error cancellation signal is received via terminal interface 56, processor 51 returns to ACT 71. Therefore, the payer cannot proceed to register the next product unless he or she takes action to register the unregistered product, that is, performs an operation to resolve the fraud of the unregistered product.

[0094] Next, we will explain the fraudulent behavior in which the settlement person registers a product, but then takes out the next product without bagging it and leaving it unregistered. This type of fraud occurs, for example, when the settlement person covers the barcode with their hand and holds it over the reading window of scanner 14, and then places the product on the floor or elsewhere without bagging it. In this case, status memory 522 stores status "22," which indicates a registration behavior, after status "21," which indicates a takeout behavior. However, status "23," which indicates a bagging behavior, is not stored next, and status "21," which indicates a takeout behavior, is stored again. That is, this corresponds to the case where status "21" is stored in the standby state of ACT52 to ACT54 in FIG. 11 (YES in ACT54).

[0095] The processor 51 proceeds to ACT81 in FIG. 14. In ACT81, the processor 51 obtains the status stored immediately before the status "21" from the status memory 522. In ACT82, the processor 51 checks whether the status stored immediately before the status "21" is "12." That is, the processor 51 checks whether the operation input data related to product registration was recognized before the settlement person's removal action. If the status "12" is stored, the products that were not bagged have been correctly registered and there is no fraud. In this case, the processor 51 proceeds to ACT83. In ACT83, the processor 51 outputs the latest product registration image stored in the list memory 523 to the self-checkout POS terminal 10. As a result, the self-checkout POS terminal 10 displays the product registration image in the registered product image display area Pa of the product registration screen SCa.

[0096] After completing the processing of ACT 83, the processor 51 returns to ACT 45 in Fig. 10. That is, the processor 51 proceeds to the same processing as when it acquires the status "21" in ACT 43. The processor 51 then executes the processing from ACT 45 onwards in the same manner as described above.

[0097] On the other hand, in ACT82, if the status stored immediately before the status "21" is not "12," the products that were not bagged are not registered. In this case, processor 51 proceeds to ACT84. Processor 51 acquires a registered behavior image in ACT84. Specifically, processor 51 acquires multiple frame images that can recognize the registered behavior stored in image memory 521 around the time of the transition to the status "22," which was stored immediately before the status "21." Processor 51 extracts an image from the acquired frame images in ACT85 that can identify the product held over the reading window by the registered behavior, as an image that identifies fraudulent activity. For example, processor 51 extracts an image of the product being held by the hand of the settlement person that is held over the reading window.

[0098] Here, the processor 51 functions as a fraud detection unit 513 by the processing of ACT82 in Fig. 14. The processor 51 also functions as an image acquisition unit 514 by the processing of ACT84 and ACT85 in Fig. 14.

[0099] After completing the processing in ACT85, the processor 51 sets the fraud category to "92" in ACT86. The fraud category "92" is information indicating fraud in which the product is not registered and not bagged. The processor 51 increments the number counter by +1 in ACT87.

[0100] The processor 51 stores the fraud data in the fraud memory 524 in ACT 88. That is, the processor 51 sets the current time measured by the clock 54 as the fraud recognition time. Then, the processor 51 stores the fraud classification determined in ACT 86, the value of the number counter updated in ACT 87, and the image identifying the fraudulent activity obtained in ACT 85, i.e., the fraud recognition image, in the fraud memory 524, in association with this recognition time.

[0101] The processor 51 outputs the fraudulent identification image and a warning message for the settler corresponding to the fraud classification "92" to the self-service POS terminal 10 as ACT89. As a result, the fraudulent identification image and the warning message for the settler are displayed in the fraud identification image display area Qa of the product registration screen SCa at the self-service POS terminal 10. The warning message notifies the settler that the product in the image is not registered, that the product that was not bagged needs to be re-registered, or the like, such as "The product is not registered. Please re-register the product that was not bagged."

[0102] The processor 51 outputs an error command as ACT90 to the self-service POS terminal 10. The self-service POS terminal 10 to which the error command is input enters an error state and becomes inoperable. The error state will not be cleared unless an attendant performs an error clearing operation.

[0103] The processor 51 outputs an error signal to the attendant terminal 20 in ACT91. The error signal includes the fraudulent identification image acquired in ACT85 and a warning message for the attendant corresponding to the fraud classification "92." The attendant terminal 20, which has received the error signal, displays the fraudulent identification image and a warning message for the attendant in the fraud identification image display area Qb on the screen SCb corresponding to the self-service POS terminal 10. The attendant terminal 20 also notifies the attendant that an error has occurred in the self-service POS terminal 10. The notification method is not limited. For example, the notification may be by lighting a lamp or sounding an error sound. After confirming the details of the fraud and the error, the attendant instructs the payer to go to the self-service POS terminal 10 and resolve the fraud. The attendant also performs an operation to resolve the error. When the error is resolved, an error resolution signal is output from the self-service POS terminal 10 to the monitoring device 50.

[0104] Here, the processor 51 functions as an output unit 515 by the processing in ACT89 and ACT91.

[0105] Processor 51 waits for an error clear signal in ACT 92. When the error clear signal is received via terminal interface 56, processor 51 returns to ACT 71 in Fig. 13. Therefore, the settlement person cannot register the next product unless they perform an operation to clear the fraud of the unregistered product that was not bagged.

[0106] Next, we will explain fraud in which the payer takes an item out of the shopping cart and then bags it without registering it. In this case, the status memory 522 stores the status "23" indicating the bagging action next to the status "21" indicating the takeout action. That is, this corresponds to the case where the status "23" is acquired (YES in ACT47) in the standby state of ACT46 to ACT48 in Figure 10.

[0107] The processor 51 proceeds to ACT61 in Figure 12. Then, the processor 51 executes the processes of ACT61 to ACT68, and further the processes from ACT71 onwards in Figure 13, in the same manner as described above. That is, the processor 51 acquires an image of the bagging activity and extracts an image that identifies fraudulent activity from the image. The processor 51 also stores the fraudulent data in the fraud memory 524. The processor 51 outputs the fraudulent recognition image and a warning message for the settlement person corresponding to the fraud category "91" to the self-checkout POS terminal 10. Furthermore, if the number of times is equal to or greater than the threshold N, the processor 51 outputs the fraudulent recognition image and a warning message for the attendant corresponding to the fraud category "91" to the attendant terminal 20.

[0108] As a result, in the self-service POS terminal 10, an image of the product that has been bagged without being registered as a fraudulent identification image and a warning message for the payer are displayed in the fraud identification image display area Qa of the product registration screen SCa. In this way, for a payer who has bagged products removed from a shopping cart without being registered, a fraudulent identification image and a warning message are displayed in the fraud identification image display area Qa of the product registration screen SCa. Furthermore, if the number of times is equal to or greater than the threshold N, a similar fraudulent identification image and a warning message for the attendant are displayed in the fraud identification image display area Qb of the screen SCb corresponding to the self-service POS terminal 10 in the attendant terminal 20. In this way, if the payer has committed fraud N or more times, a report is made to the attendant via the attendant terminal 20.

[0109] Thereafter, processor 51 waits for the next status data to be status "22," status "24," or status "21." If status "22" is acquired, processor 51 stops outputting the fraudulent recognition image and warning message. In other words, if the person making the payment corrects the fraud of not registering the product, subsequent product registration becomes possible. However, if the person removes the next product from the shopping cart without correcting the fraud, self-checkout POS terminal 10 enters an error state.

[0110] Finally, we will explain fraud where the customer leaves the checkout area without paying for the purchased items. This type of fraud is discovered when the next status obtained is status "24" (YES in ACT44, ACT48, ACT54 or ACT74) in the standby state of ACT43 or ACT44 in Fig. 10, or in the standby state of ACT46 to ACT48 in the same figure, or in the standby state of ACT52 to ACT54 in Fig. 11, or in the standby state of ACT72 to ACT74 in Fig. 13.

[0111] Processor 51 proceeds to ACT101 in FIG. 10. Processor 51 obtains the status stored immediately before status "24" from state memory 522. Processor 51 checks whether the status stored immediately before status "24" is "14" or "15" as ACT102. That is, processor 51 checks whether operation input data related to the completion of transaction or the cancellation of registration is recognized before the exit action. If operation input data related to the completion of transaction or the cancellation of registration is recognized before the exit action, the exit action is a legitimate action. Processor 51 terminates its function as fraud detection unit 513.

[0112] In contrast, if the status stored immediately before status "24" is not "14" or "15," i.e., if the operational input data related to the completion of the transaction or the cancellation of registration is not recognized, the exit behavior is considered fraudulent. Processor 51 acquires an exit behavior image in ACT 103. Specifically, processor 51 acquires multiple frame images that recognize the exit behavior stored in image memory 521 around the time of the transition to status "24." Processor 51 then extracts an image from the acquired frame images that can identify the person making the payment as an image that identifies fraudulent activity in ACT 104. For example, processor 51 extracts an image that shows the face of the person making the payment.

[0113] In ACT 105, processor 51 sets the fraud category to "99." Fraud category "99" is information indicating fraud involving leaving without paying. In ACT 106, processor 51 counts up the number counter by +1. Then, in ACT 107, processor 51 stores the fraud data in fraud memory 524. That is, processor 51 sets the current time kept by clock 54 as the fraud recognition time. Then, processor 51 associates the fraud category determined in ACT 105, the value of the number counter updated in ACT 106, and the image identifying the fraudulent activity obtained in ACT 104, i.e., the fraud recognition image, in fraud memory 524.

[0114] The processor 51 outputs an error command to the self-service POS terminal 10 in ACT108. The processor 51 also outputs an error signal to the attendant terminal 20 in ACT109. The error signal includes the fraudulent recognition image acquired in ACT104 and a warning message for the attendant corresponding to the fraud category "99." The warning message alerts the attendant, for example, saying, "There is a customer attempting to leave without paying." The processor 51 that has output the error command and error signal terminates the function as the fraud detection unit 513.

[0115] [Explanation of the effects of monitoring devices] As described above in detail, the monitoring device 50 has the effect of preventing fraudulent acts by customers that may occur at the self-service POS terminal 10.

[0116] For example, if a payment maker commits fraud by bagging purchased items without registering them, a fraudulent image and a warning message are displayed on the touch panel 11 of the self-service POS terminal 10. The fraudulent image is an image that can identify the purchased items that the payment maker bagged without registering them. The warning message instructs the payment maker to re-register the unregistered items. Displaying the fraudulent image together with the warning message can have a deterrent effect on the fraudulent acts described above. In other words, displaying the fraudulent image together with the warning message on the touch panel 11 may alert other customers or store clerks to the fact that fraud has occurred. Moreover, because the fraudulent image is an image that can identify unregistered items, it is also easy to identify which items the fraudulent act was committed against. This makes it more difficult for the payment maker to commit fraud.

[0117] Furthermore, if the person making the payment ignores the warning message and proceeds to register the next product, the self-service POS terminal 10 is rendered inoperable and a warning is issued to the attendant terminal 20. Therefore, the fraudulent acts described above will not be overlooked.

[0118] For example, a fraudulent act may occur in which the payer only registers the purchased items but does not bag them. In this case, a fraudulent image and a warning message are displayed on the touch panel 11 of the self-service POS terminal 10. The fraudulent image is an image that can identify the purchased items that the payer registered. Therefore, similar to the above, this can have a deterrent effect on fraudulent acts.

[0119] Furthermore, if the payer commits fraudulent acts more than the threshold number N, a similar fraud recognition image and a warning message are displayed on the display device of the attendant terminal 20. Therefore, the attendant can notice the fraudulent acts and alert the payer, which has a significant deterrent effect on fraudulent acts.

[0120] Furthermore, if a person attempts to leave the checkout area without paying, a fraudulent recognition image showing, for example, the person's face is displayed on the display device of attendant terminal 20. Therefore, even if the checkout area is crowded with many people, an attendant can easily identify a person who is leaving fraudulently.

[0121] [Variations] In the above embodiment, an example has been given in which each self-service POS terminal 10 is equipped with the monitoring device 50. One monitoring device 50 may be connected to two or more self-service POS terminals 10, and the processor 51 of the monitoring device 50 may have the functions of an operation recognition unit 511, a behavior recognition unit 512, a fraud detection unit 513, an image acquisition unit 514, and an output unit 515 for each of the two or more self-service POS terminals 10.

[0122] The camera 60 does not have to be attached to the self-service POS terminal 10. For example, it may be attached to the ceiling of the checkout area. Furthermore, the actions of the person making the payment may be recognized from images captured by two or more cameras.

[0123] The actions recognized by the action recognition unit 512 are not limited to taking out, registering, bagging, and leaving. For example, a series of actions such as taking out an item from a shopping cart and bagging it without registering it, or handing it over to a companion, may be recognized as fraudulent actions. In this case, an image that identifies the item may be extracted from the image in which fraudulent action has been recognized, and the image may be output to the self-POS terminal 10 or the attendant terminal 20. The image may also be output to another terminal, such as an information communication terminal carried by a store clerk. The action of the person making the payment or a person other than the person making the payment hiding the camera 60 with their hand, arm, etc. may also be recognized.

[0124] In the above embodiment, the function of the output unit 515 is to output the fraud recognition image together with a warning message, but the function of the output unit 515 is not limited to this. For example, even if only the fraud recognition image is output to be displayed on the self-service POS terminal 1, the fraud prevention effect can be achieved.

[0125] In the above embodiment, the fraud detection image is output each time fraud is detected, but the timing of image output is not limited to this. For example, the fraud detection image may be output when the settlement person touches the accounting key Bx, that is, when an instruction to start accounting is given. In this case, if the determination is YES in ACT 74 of Figure 13, the processing of ACT 76 to ACT 78 may be omitted and the processing may proceed to ACT 45 of Figure 10.

[0126] In the above embodiment, the processor 51 recognizes the start of use when the start key on the payment start screen is touched. For example, a human presence sensor that can detect a person standing in front of the self-service POS terminal 10 may be provided, and the processor 51 may recognize the start of use when the human presence sensor detects a person. In this case, the processor 51 may recognize that an exit action has been performed when the human presence sensor no longer detects a person.

[0127] In the above embodiment, it has been described that the monitoring device 50 may be located either externally or internally to the self-service POS terminal 10. In another embodiment, the self-service POS terminal 10 may incorporate the functions of the monitoring device 50. That is, the payment device (self-service POS terminal 10) may include: behavior recognition means (behavior recognition unit 512) that recognizes the behavior of a payment person; operation recognition means (operation recognition unit 511) that recognizes operation inputs by the payment person; fraud detection means (fraud detection unit 513) that detects fraudulent activity by the payment person based on the recognition results by the behavior recognition means and the operation recognition means; image acquisition means (image acquisition unit 514) that acquires an image identifying the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures the payment person operating the payment device; and output means (output unit 515) that outputs the image identifying the fraudulent activity.

[0128] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0129] 10...self-service POS terminal, 11...touch panel, 14...scanner, 20...attendant terminal, 30...store server, 40...communication network, 50...monitoring device, 51...processor, 52...main memory, 53...auxiliary storage device, 54...clock, 55...camera interface, 56...terminal interface, 57...network interface, 60...camera, 100...checkout system, 102...first product placement stand, 103...second product placement stand, 511...operation recognition unit, 512...behavior recognition unit, 513...fraud detection unit, 514...image acquisition unit, 515...output unit, 521...image memory, 522...status memory, 523...list memory, 524...fraud memory, Qa, Qb...fraud identification image display area.

Claims

1. a behavior recognition means for recognizing a behavior of a payer with respect to the payment device; an operation recognition means for recognizing an operation input to the payment device by the payment person; a fraud detection means for detecting fraudulent activity by the settling person based on the recognition results by the behavior recognition means and the operation recognition means; an image acquisition means for acquiring an image identifying the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures an image of a settlement person operating the settlement device; an output means for outputting an image that identifies the fraudulent activity; A monitoring device comprising:

2. The monitoring device according to claim 1 , wherein the behavior recognition means recognizes the behavior of the person making the payment based on an image captured by the camera.

3. The behavior recognition means recognizes at least a take-out behavior of the settlement person picking up a product and a bagging behavior of the settlement person packing the product in a bag; the operation recognition means recognizes at least an operation input for registering a commodity to the payment device; the fraud detection means detects the fraudulent activity when the operation recognition means fails to recognize an operation input for registering the product while the behavior recognition means recognizes the picking action and the bagging action, 3. The monitoring device according to claim 2, wherein the image acquisition means detects an image including the products packed in a bag by the bagging action from the images taken by the camera used to recognize the bagging action of the settlement person.

4. The behavior recognition means recognizes at least a registration behavior of the settling person registering the product he or she has picked up in the settling device, the operation recognition means recognizes at least an operation input for registering a commodity to the payment device; the fraud detection means detects the fraudulent activity when the behavior recognition means recognizes the registration behavior but the operation recognition means fails to recognize the operation input for product registration, The monitoring device according to claim 2 , wherein the image acquisition means detects, from the images captured by the camera used to recognize the payment person's registration behavior, an image including a commodity held up to the payment device as part of the registration behavior.

5. 5. The monitoring device according to claim 1, wherein the output means displays an image specifying the fraudulent activity on a display device of the settlement device.

6. Computer, behavior recognition means for recognizing the behavior of a payer with respect to the payment device; an operation recognition means for recognizing an operation input to the payment device by the payment person; a fraud detection means for detecting fraudulent activity by the settling person based on the recognition results by the behavior recognition means and the operation recognition means; an image acquisition means for acquiring an image identifying the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures an image of a settlement person operating the settlement device; and an output means for outputting an image identifying the fraudulent activity; A program to function as a

7. A behavior recognition means for recognizing the behavior of a payer; an operation recognition means for recognizing an operation input by the settler; a fraud detection means for detecting fraudulent activity by the settling person based on the recognition results by the behavior recognition means and the operation recognition means; an image acquisition means for acquiring an image identifying the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures an image of the settler; an output means for outputting an image that identifies the fraudulent activity; A payment device comprising:

8. Computer, a behavior recognition means for recognizing the behavior of the payment user; an operation recognition means for recognizing an operation input by the settler; a fraud detection means for detecting fraudulent activity by the settling person based on the recognition results by the behavior recognition means and the operation recognition means; an image acquisition means for acquiring an image that identifies the fraudulent activity detected by the fraud detection means from an image captured by a camera that captures the settler; and an output means for outputting an image identifying the fraudulent activity; A program to function as a

Images