Monitoring device and its program

Abstract

Prevents fraudulent behavior such as leaving without paying while reducing the risk of false positive reports. [Solution] The monitoring device comprises a customer detection means, an item detection means, and a control means. The customer detection means detects a customer at the operating position of a checkout machine where transaction-related operations are performed. The item detection means detects items placed on the checkout machine by the customer. The control means controls the device so that, if the customer who was detected by the customer detection means is no longer detected and no items are detected by the item detection means between the start and end of the customer's transaction-related operations on the checkout machine, an alert is issued that the customer has left without completing the transaction.

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a monitoring device and a program therefor. [Background technology]

[0002] In recent years, full-self-service POS (Point of Sales) terminals, so-called self-service POS terminals, have become widespread in retail stores such as supermarkets and convenience stores. The introduction of self-service POS terminals allows customers to operate the terminal themselves to perform all accounting tasks, from registering purchased items to paying for them, which has the advantage of reducing manpower and reducing personnel costs.

[0003] On the other hand, when self-service POS terminals are introduced, there are concerns about fraudulent behavior, such as customers leaving without paying.To address this issue, a monitoring device has been developed that uses a camera to record the actions of customers operating the self-service POS terminal, analyzes the footage using AI (artificial intelligence), and reports any fraudulent behavior detected, such as customers leaving without paying.

[0004] However, such monitoring devices may mistakenly detect fraudulent behavior as a customer leaving without paying if the customer temporarily leaves the self-service POS terminal during the checkout process, for example to get a shopping bag or to return to the sales floor to get an item they forgot to buy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2019-096035 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem that the embodiments of the present invention aim to solve is to provide a monitoring device that can efficiently and effectively prevent or deter the fraudulent act of leaving without paying while reducing the risk of false positive reports. [Means for solving the problem]

[0007] In one embodiment, the monitoring device includes a customer detection means, an item detection means, and a control means. The customer detection means detects a customer at an operating position of a checkout machine where a checkout operation is performed. The item detection means detects an item placed on the checkout machine by the customer. The control means controls the device so that, if the customer who was detected by the customer detection means is no longer detected and no item is detected by the item detection means between the start and end of the customer's checkout operation on the checkout machine, an alert is issued that the customer has left without completing the checkout. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of a checkout system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing the positional relationship between the self-service POS terminal and the purchaser who is its operator. [Figure 3] FIG. 3 is a schematic diagram showing the imaging area of ​​the camera. [Figure 4] FIG. 4 is a schematic diagram showing an example layout of a checkout area. [Figure 5] FIG. 5 is a block diagram showing the main circuit configuration of the monitoring device. [Figure 6] FIG. 6 is a block diagram showing the main functional configuration of the processor of the monitoring device. [Figure 7] FIG. 7 is a flowchart showing the main steps of information processing executed by the processor of the monitoring device in accordance with the monitoring program. [Figure 8] FIG. 8 is a flowchart showing the main steps of information processing executed by the processor of the monitoring device in accordance with the monitoring program. [Figure 9]FIG. 9 is a flowchart showing the main steps of information processing executed by the processor of the monitoring device in accordance with the monitoring program. [Figure 10] FIG. 10 is a schematic diagram showing a schematic configuration of a checkout system according to another embodiment. [Figure 11] FIG. 11 is an explanatory diagram of an image processing area in another embodiment. [Figure 12] FIG. 12 is an explanatory diagram of an image processing area in another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of a monitoring device that can efficiently and effectively prevent or deter fraudulent behavior such as leaving without paying while reducing the risk of false positive notifications will be described with reference to the drawings.

[0010] [Explanation of the self-checkout system] First, an embodiment of a checkout system 100 to which a monitoring device 50 is applied will be described with reference to FIGS.

[0011] 1 is a schematic diagram showing the overall configuration of a checkout system 100. The checkout system 100 includes a plurality of self-service POS terminals 10, an attendant terminal 20, and a store server 30. The checkout system 100 connects each of the self-service POS terminals 10, the attendant terminal 20, and the store server 30 via a communication network 40 such as a LAN (Local Area Network). The store server 30 provides various services related to the checkout system 100 to each of the self-service POS terminals 10 and the attendant terminal 20 connected via the communication network 40.

[0012] A plurality of self-service POS terminals 10 are installed at the checkout area RE (see FIG. 4) of the store, and are operated by consumers who have finished shopping, i.e., purchasers. A purchaser places purchased items in a shopping cart or the like in a sales area where items are displayed and moves to the checkout area RE. The purchaser operates the self-service POS terminal 10 to perform the checkout process by himself, from registering the purchased items to paying for them. The self-service POS terminal 10 is an example of a checkout machine.

[0013] The attendant terminal 20 is installed in a monitoring station located near the exit of the checkout area RE. The attendant terminal 20 displays, on a display device, a list of purchased items registered for each self-service POS terminal 10. If an error occurs in the self-service POS terminal 10, the attendant terminal 20 notifies the customer of the error. The store clerk at the monitoring station, i.e., the attendant, monitors the checkout area RE based on the information displayed and notified on the attendant terminal 20. Although FIG. 1 shows one attendant terminal 20, multiple attendant terminals 20 may be connected to the communication network 40.

[0014] Each self-service POS terminal 10 includes a monitoring device 50. The monitoring device 50 may be located either externally or internally within the self-service POS terminal 10. The monitoring device 50 is a device for monitoring a customer operating the self-service POS terminal 10. Specifically, the monitoring device 50 monitors a customer operating the self-service POS terminal 10 to perform a transaction based on information such as images captured by a camera 60 installed near the self-service POS terminal 10. The monitoring device 50 then issues an alert if it detects fraudulent behavior by a customer. Fraudulent behavior, for example, is leaving the checkout area without paying for the purchased items. The alert is sent to the attendant terminal 20. Each monitoring device 50 is connected to the attendant terminal 20 via the communication network 40 and outputs an alert signal to the attendant terminal 20. Each monitoring device 50 may also output an alert signal to the attendant terminal 20 without using the communication network 40. The destination of the alert is not limited to the attendant terminal 20. The destination of the alert may include the self-POS terminal 10. Alternatively, the destination of the alert may be another device, such as a communication terminal such as a smartphone carried by the store clerk.

[0015] Fig. 2 is a schematic diagram showing the positional relationship between the self-service POS terminal 10 and the purchaser SH who is its operator. Fig. 2 shows the self-service POS terminal 10 as viewed from above. As shown in the figure, the self-service POS terminal 10 has a terminal housing 11 sandwiched between it, with a shopping basket SB mounting stand 12 on one side and an automatic change dispenser 13 on the other side. Operation devices such as a touch panel 14 and a scanner 15 are provided on the top of the terminal housing 11, and a receipt printer 16 and the like are provided above the automatic change dispenser 13. In addition, the area in front of the touch panel 14 and scanner 15 at the top of the terminal housing 11 is a space SP for bagging.

[0016] The customer SH moves to the checkout area RE and stands in front of the terminal housing 11 so that the table 12 is located on the right side and the automatic change dispenser 13 is located on the left side. The customer SH then prepares a reusable bag or a plastic bag in the space SP, operates the touch panel 14 to select a reusable bag or a plastic bag, and registers the data of the purchased items in the self-service POS terminal 10. That is, the customer SH removes each item from the shopping cart SB placed on the table 12 and scans the barcode of each item with the scanner 15. If the purchased item does not have a barcode, the customer SH operates the touch panel 14 to display a list of items without barcodes and selects the purchased item from that list. The customer SH places the items whose data has been registered in the self-service POS terminal 10 into the reusable bag or plastic bag. After registering the data of all the purchased items in this way, the customer SH operates the touch panel 14 to select a payment method. If the customer SH selects cash payment, for example, the customer SH inserts cash into the automatic change dispenser 13. Thus, when payment is completed, a receipt is issued from the receipt printer 16, and the purchaser SH receives the receipt and leaves the checkout area RE with the bag or shopping bag containing the purchased items.

[0017] In order to monitor the transaction behavior of such customers, a camera 60 is attached above the self-service POS terminal 10. Fig. 3 is a schematic diagram showing the image capture area ARa of the camera 60. As shown in the figure, the image capture area ARa of the camera 60 includes the terminal housing 11, the stand 12, and the automatic change dispenser 13 of the self-service POS terminal 10, as well as the space in front of them where customers enter and exit. The monitoring device 50, which captures an image of the image capture area ARa from the camera 60, defines an area of ​​the image capture area ARa that corresponds to the operation position of the customer SH, which is on the front side of the terminal housing 11, as a person detection area ARb. The monitoring device 50 also defines an item detection area ARc as an area that includes the terminal housing 11 on which the customer SH may place their own bag or a shopping bag, the platform 12 on which the customer SH may place a shopping basket SB, and the automatic change machine 13 on which the customer SH may place their personal belongings. Note that the person detection area ARb and the item detection area ARc may partially overlap.

[0018] 4 is a schematic diagram showing an example layout of a checkout area RE. In this embodiment, the checkout area RE has a rectangular space, with one side of the space being an entrance ENT and the other side being an exit EXT. On both sides of the space between the entrance ENT and the exit EXT, multiple self-POS terminals 10 (four in the figure) are arranged facing each other. In addition, a monitoring station is set up near the exit EXT, and an attendant terminal 20 is installed at this monitoring station so that an attendant AT can monitor the checkout area RE.

[0019] In a checkout area RE with such a layout, a customer SH who has finished shopping in the sales area enters the checkout area RE through the entrance ENT and stands in front of an empty self-service POS terminal 10. The customer SH then operates the self-service POS terminal 10 to register the purchased items, pays for them, and then exits through the exit EXT.

[0020] At such a checkout counter RE, a customer SH may commit fraud by leaving the exit EXT without paying, a so-called "leaving without paying" behavior. As described above, the monitoring device 50 detects such leaving without paying. Specifically, the monitoring device 50 recognizes this as a leaving without paying behavior when the customer SH leaves the person detection area ARb before completing payment at the self-service POS terminal 10 where the customer SH initiated registration of the purchased items through self-service operation. However, the customer SH may also leave the self-service POS terminal 10 during checkout as legitimate behavior. For example, this may include going to get a plastic bag prepared nearby at the entrance ENT or returning to the sales floor from the entrance ENT because they forgot to buy an item. The monitoring device 50 does not issue an alert for such legitimate behavior. An embodiment of the monitoring device 50 that operates in this manner will be described below with reference to FIGS.

[0021] [Configuration of monitoring device] 5 is a block diagram showing the main circuit configuration of a monitoring device 50. The monitoring device 50 includes circuit components such as a processor 51, a main memory 52, an auxiliary storage device 53, a clock 54, a terminal interface 55, a camera interface 56, and a network interface 57, as well as a system transmission path 58. The system transmission path 58 includes an address bus, a data bus, a control signal line, etc. The monitoring device 50 configures a computer by connecting each circuit component to the system transmission path 58.

[0022] The processor 51 corresponds to the central part of a computer. The processor 51 controls each part to realize various functions of the monitoring device 50 in accordance with an operating system or an application program (application software). The processor 51 is, for example, a CPU (Central Processing Unit).

[0023] The main memory 52 corresponds to the main storage portion of the computer. The main memory 52 includes a nonvolatile memory area and a volatile memory area. The main memory 52 stores an operating system or application programs in the nonvolatile memory area. The main memory 52 may store data required for the processor 51 to execute processes for controlling each part in either the nonvolatile or volatile memory area. The main memory 52 uses the volatile memory area as a work area where data is rewritten by the processor 51 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0024] The auxiliary storage device 53 corresponds to the auxiliary storage portion of the computer. For example, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 53. The auxiliary storage device 53 stores data used by the processor 51 when performing various processes, or data created by the processes of the processor 51. The auxiliary storage device 53 may also store the application programs described above.

[0025] The clock 54 keeps track of the date and time. The monitoring device 50 processes the date and time kept by the clock 54 as the current date and time.

[0026] The terminal interface 55 connects to the self-service POS terminal 10. The terminal interface 55 controls two-way data communication between the self-service POS terminal 10. The connection between the monitoring device 50 and the self-service POS terminal 10 may be wired or wireless.

[0027] The camera interface 56 connects to the camera 60 provided in the self-service POS terminal 10. The camera interface 56 outputs a control signal to the camera 60 to control the operation of the camera 60. The camera 60 performs a photographing operation in accordance with this control signal. The camera interface 56 then captures image data of the photographing area ARa photographed by the camera 60.

[0028] The network interface 57 is connected to the communication network 40. The network interface 57 performs data communication with, for example, the attendant terminal 20 via the communication network 40. The network interface 57 can also perform data communication with other devices connected to the communication network 40.

[0029] [Functional description of monitoring device] 6 is a block diagram showing the main functional configuration of the processor 51 of the monitoring device 50. As shown in the figure, the processor 51 has the functions of an image storage means 511, an image comparison means 512, a purchaser detection means 513, an item detection means 514, a status recognition means 515, a prohibition means 516, a determination means 517, and a control means 518.

[0030] The image storage unit 511 stores an image of the self-service POS terminal 10 taken from above by the camera 60 when the customer SH is not present at the operation position of the self-service POS terminal 10 where the customer SH performs the transaction-related operations. Transaction-related operations include operations from registering the purchased items to paying the price. For example, when the self-service POS terminal 10 is in its default (standby) state, the self-service POS terminal 10 determines that the transaction-related operation has begun by detecting the customer SH's selection of a personal bag or a shopping bag, the selection of the "Start Shopping" button, the registration of information about the first item, or the acquisition of information about the membership card. The operation position is included in the person detection area ARb described above. Therefore, the image storage unit 511 stores an image of the self-service POS terminal 10 taken from above by the camera 60 when the customer SH is not present in the person detection area ARb. Hereinafter, the image stored by the image storage unit 511 will be referred to as the "initial image." The initial image may be stored in the main memory 52 or the auxiliary storage device 53.

[0031] The image comparison means 512 has a function of comparing the image captured by the camera 60 after the purchaser SH has started a transaction-related operation on the self-service POS terminal 10 with the initial image stored in the image storage means 511. The image comparison means 512 has a first comparison means 5121 that compares the captured image with the initial image for a person detection area ARb that includes the operation position of the self-service POS terminal 10, and a second comparison means 5122 that compares the captured image with the initial image for an item detection area ARc that includes a location where items such as the shopping basket SB and personal bags of the self-service POS terminal 10 are placed. That is, the first comparison means 5121 compares the captured image with the initial image for the person detection area ARb. The second comparison means 5122 compares the captured image with the initial image for the item detection area ARc.

[0032] The customer detection means 513 is a function that detects a customer SH who is present at the operation position of the self-service POS terminal 10 where transaction-related operations are performed. The customer detection means 513 detects the customer SH based on the image comparison results obtained by the first comparison means 5121. The customer is not shown in the initial image compared by the first comparison means 5121. However, when the customer SH enters the person detection area ARb, the customer SH is shown in the captured image. When the customer detection means 513 detects a captured image in which the customer SH is shown in the person detection area ARb through comparison of the initial image with the captured image by the first comparison means 5121, it processes the image as having detected the customer SH. On the other hand, when the customer detection means 513 no longer detects a captured image in which the customer SH is shown in the person detection area ARb, that is, when the captured image in the person detection area ARb becomes the same as the initial image, it processes the image as having no longer detected the customer SH.

[0033] The item detection means 514 has a function of detecting items placed on the self-service POS terminal 10 by the customer SH. Examples of items include a reusable bag or a plastic bag placed in the bagging space SP of the terminal housing 11, a shopping basket SB placed on the platform 12, and personal belongings placed on the automatic change dispenser 13. The item detection means 514 detects items based on the results of image comparison by the second comparison means 5122. In principle, the initial image compared by the second comparison means 5122 does not show the above-mentioned items. However, when the customer SH prepares a reusable bag or a plastic bag in the bagging space SP, places a shopping basket SB on the platform 12, or places personal belongings on the automatic change dispenser 13, the item is shown in the captured image. When the item detection means 514 detects an image showing an item in the item detection area ARc by comparing the initial image with the captured image by the second comparison means 5122, the item detection means 514 processes the image as having detected the item. On the other hand, if no article is shown in the captured image, that is, if the captured image is in the same state as the initial image for the article detection area ARc, the article detection means 514 processes the captured image as not having detected an article.

[0034] The status recognition means 515 is a function that recognizes the status of the self-service POS terminal 10. The status of the self-service POS terminal 10 includes a default status before the start of a transaction, a registration status after a command to start registration has been issued, and a transaction status after a command to start payment has been issued. The status of the self-service POS terminal 10 transitions from the default status to a registration status, and then transitions to a transaction status before returning to the default status. The self-service POS terminal 10 outputs a status signal indicating the current status to the attendant terminal 20. The monitoring device 50 acquires this status signal. The monitoring device 50 may acquire the status signal via the attendant terminal 20 or may acquire it directly from the self-service POS terminal 10. The status recognition means 515 recognizes the status of the self-service POS terminal 10 based on the status signal.

[0035] The self-service POS terminal 10 can be recognized by a status signal to have a store clerk called state. When the store clerk call button displayed on the screen of the touch panel 14 is touched while the self-service POS terminal 10 is in the registration or transaction state, the self-service POS terminal 10 transitions to the store clerk called state. The state recognition means 515 can also recognize the store clerk called state. The store clerk called state is canceled by a predetermined operation on the self-service POS terminal 10 by a store clerk, for example, an attendant.

[0036] The prohibition means 516 has a function that prohibits operation of the self-checkout POS terminal 10 until the customer SH is detected again by the customer detection means 513 when the following first and second events occur. The first event is when the customer SH disappears from the person detection area ARb between the start and end of a transaction-related operation by the customer SH on the self-checkout POS terminal 10. The second event is when an item such as a reusable shopping bag, a shopping bag, a shopping basket, or personal belongings is placed in the item detection area ARc. That is, when the status of the self-checkout POS terminal 10 acquired by the status recognition means 515 is in the registration or transaction state, and the customer SH who was detected by the customer detection means 513 is no longer detected, but items are still detected by the item detection means 514, the prohibition means 516 outputs an operation prohibition signal to the self-checkout POS terminal 10. Thereafter, when the customer SH is detected again by the customer detection means 513, the prohibition means 516 stops the operation prohibition signal. Incidentally, in the self-service POS terminal 10, while the operation prohibition signal is being received, operation devices such as the touch panel 14 and the scanner 15 become inoperable.

[0037] The determination unit 517 has a function of determining whether an item detected by the item detection unit 514 is an item that is not subject to leaving detection. Items that a customer determines to be unnecessary may be left behind as residue in the bagging space SP of the self-service POS terminal 10, the placement table 12, or the top surface of the automatic change dispenser 13. For example, if a customer picks up cutlery such as a knife, fork, spoon, or disposable chopsticks provided around the self-service POS terminal 10 but then decides that they are unnecessary and places them on the automatic change dispenser 13, or if the customer does not take the receipt issued by the receipt printer 16 home with them, the cutlery, receipt, etc. are left behind as residue. In this embodiment, such items that may become residue are defined as items that are not subject to leaving detection. The determination unit 517 determines whether an item detected by the item detection unit 514 is an item that is not subject to leaving detection, such as cutlery or a receipt. For example, the determination means 517 stores in advance an image of an item that is not subject to leaving detection, and calculates the similarity by comparing that image with the image of the item detected by the item detection means 514. Then, when the similarity is equal to or greater than a predetermined threshold, the determination means 517 determines that the item detected by the item detection means 514 is not subject to leaving detection.

[0038] The control means 518 has a function to control the attendant terminal 20 to notify the attendant terminal 20 of a customer's departure without making a payment when the first event described above and the third event described below occur. The third event is an event in which no item is placed within the item detection area ARc. That is, when the status of the self-service POS terminal 10 acquired by the status recognition means 515 is in the registration or transaction in progress state, the control means 518 outputs a notification signal of customer's departure without making a payment if the customer SH, who was detected by the customer detection means 513, is no longer detected and no item is detected by the item detection means 514. Furthermore, even if an item is detected by the item detection means 514, the control means 518 outputs the notification signal of customer's departure without making a payment if the determination means 517 determines that the item is not subject to departure detection. The notification signal is output to, for example, the attendant terminal 20. Upon receiving the notification signal, the attendant terminal 20 notifies the attendant terminal 20 of customer's departure without making a payment. For example, the attendant terminal 20 displays an image on a monitor to notify the customer that they have left without paying. Alternatively, the attendant terminal 20 emits an alarm sound from a speaker to notify the customer that they have left without paying. The notification signal may be sent to other devices such as the self-service POS terminal 10 and a communication terminal carried by the store clerk.

[0039] [Explanation of information processing by monitoring equipment] The functions of the monitoring device 50 as an image storage means 511, an image comparison means 512, a purchaser detection means 513, an item detection means 514, a status recognition means 515, a prohibition means 516, a judgment means 517 and a control means 518 are realized by the processor 51 executing information processing in accordance with the monitoring program.

[0040] The monitoring program is a type of application program stored in the main memory 52 or the auxiliary storage device 53. There are no particular limitations on the method for installing the monitoring program in the main memory 52 or the auxiliary storage device 53. The monitoring program can be recorded on a removable recording medium, or can be distributed via communication over a network and installed in the main memory 52 or the auxiliary storage device 53. The recording medium can be in any form, such as a CD-ROM or memory card, as long as it can store the program and is readable by the device.

[0041] Next, the information processing procedure executed by the processor 51 of the monitoring device 50 in accordance with the monitoring program will be described with reference to the flowcharts of FIGS.

[0042] The processor 51 of the monitoring device 50 in the default state stores an initial image in the first image memory as ACT1. As described above, the initial image is an image taken by the camera 60 from above of the self-service POS terminal 10 when the customer SH is not present at the operation position. The first image memory is part of the volatile memory area in the main memory 52. ​​The initial image is saved in advance in the auxiliary storage device 53 and is stored in the first image memory at the time of processing ACT1. Alternatively, the initial image may be captured by capturing an image taken by the camera 60 when the monitoring device 50 enters the default state and stored in the first image memory as the initial image.

[0043] After storing the initial image, processor 51 proceeds to ACT2. In ACT2, processor 51 acquires the current image captured by camera 60, the so-called captured image. Then, in ACT3, processor 51 compares the captured image with the initial image. That is, processor 51 compares the initial image, in which no person is captured in the person detection area ARb, with the current captured image. Then, in ACT4, processor 51 checks whether a person is captured in the person detection area ARb of the captured image. If no person is captured in the captured image, that is, if there is no difference between the images of the person detection area ARb in the initial image and the captured image, processor 51 returns to ACT2. Processor 51 executes the processes from ACT2 onwards in the same manner as described above.

[0044] In ACT4, when a person is detected from the captured image, the processor 51 proceeds to ACT5. In ACT5, the processor 51 checks whether registration has started in the self-service POS terminal 10. That is, the processor 51 checks whether the status of the self-service POS terminal 10 has transitioned from the default state to a registering state.

[0045] If the status of the self-service POS terminal 10 is in the default state, the processor 51 proceeds to ACT6. The processor 51 acquires a captured image in ACT6. Then, the processor 51 compares the captured image with the initial image in ACT7. In this comparison, the processor 51 checks whether a person is shown in the person detection area ARb of the captured image in ACT8. If a person is shown in the captured image, the processor 51 returns to ACT5. The processor 51 executes the processes from ACT5 onwards in the same manner as described above.

[0046] On the other hand, if a person is not detected in the captured image in ACT 8, that is, if there is no difference between the images of the person detection area ARb in the initial image and the captured image, the processor 51 returns to ACT 2. The processor 51 executes the processes from ACT 2 onwards in the same manner as described above.

[0047] In this way, the processor 51 repeatedly executes the processes of ACT2 to ACT4 when there is no person in the person detection area ARb of the self-service POS terminal 10. When a person enters the person detection area ARb, the processor 51 repeatedly executes the processes of ACT5 to ACT8 until the status of the self-service POS terminal 10 transitions to a registration in progress state. Here, if the person leaves the person detection area ARb without performing an operation to start registration on the self-service POS terminal 10, the processor 51 will again repeat the processes of ACT2 to ACT4.

[0048] On the other hand, if the person who enters the person detection area ARb is a purchaser SH and selects a personal bag or a plastic bag on the self-service POS terminal 10 in the default state, or selects the "Start Shopping" button displayed on the screen, the state of the self-service POS terminal 10 transitions to a registration in progress state. In response to this transition, the processor 51 of the monitoring device 50 recognizes that registration has begun in the self-service POS terminal 10. The processor 51 proceeds from ACT5 to ACT9. In ACT9, the processor 51 stores a person image in the second image memory. The person image is an image captured by the camera 60 immediately after the state of the self-service POS terminal 10 transitions from the default state to the registration in progress state. This captured image shows a person who is in the person detection area ARb. The second image memory is part of the volatile memory area in the main memory 52.

[0049] After storing the person image, the processor 51 proceeds to ACT10. The processor 51 acquires a captured image in ACT10. Then, the processor 51 compares the captured image with the initial image in ACT11. In this comparison, the processor 51 checks whether a person is shown in the person detection area ARb of the captured image in ACT12. If a person is shown in the captured image, the processor 51 proceeds to ACT13. The processor 51 checks whether the state of the self-service POS terminal 10 has transitioned to a store clerk calling state in ACT13. Here, if the state of the self-service POS terminal 10 has transitioned to a store clerk calling state, the processor 51 waits for the store clerk calling state to be released.

[0050] If the state of the self-service POS terminal 10 has not transitioned to the store clerk calling state, or if the store clerk calling state is released, the processor 51 proceeds to ACT14. In ACT14, the processor 51 checks whether the transaction has been completed at the self-service POS terminal 10. That is, the processor 51 checks whether the state of the self-service POS terminal 10 has transitioned from the transaction in progress state to the default state. If the state of the self-service POS terminal 10 is in the registration or transaction in progress state, that is, if the transaction has not been completed at the self-service POS terminal 10, the processor 51 returns to ACT10. The processor 51 executes the processing from ACT10 onwards in the same manner as described above.

[0051] In this way, while the person is displayed in the person detection area ARb of the captured image, the processor 51 of the monitoring device 50 that has stored the person image waits for the transaction to be completed at the self-service POS terminal 10. If the state of the self-service POS terminal 10 transitions to a store clerk calling state during that time, the processor 51 waits until the store clerk calling state is released.

[0052] When the state of the self-checkout POS terminal 10 transitions to the default state in ACT14, the processor 51 proceeds to ACT 15. The processor 51 clears the person image stored in the second image memory in ACT 15. Thereafter, the processor 51 returns to ACT1.

[0053] On the other hand, if in ACT12 a person is not detected in the captured image, that is, if there is no difference between the images of the person detection area ARb in the initial image and the captured image, the processor 51 proceeds to ACT21 in Fig. 8. In ACT21, the processor 51 checks whether an object is shown in the object detection area ARc of the captured image. If an object is not shown, that is, if there is no difference between the images of the object detection area ARc in the initial image and the captured image, the processor 51 proceeds to ACT23.

[0054] On the other hand, if an object is shown in the captured image, the processor 51 proceeds to ACT 22. In ACT 22, the processor 51 checks whether the object is an object that is not a detection target. For example, the processor 51 calculates the similarity between the image of the object shown in the captured image and the image of an object defined as an object that is not a detection target, and checks whether the similarity is equal to or greater than a predetermined threshold. If the similarity is equal to or greater than the threshold, the processor 51 determines that the object shown in the captured image is an object that is not a detection target. The processor 51 proceeds to ACT 23.

[0055] In this way, when processor 51 confirms that no person is shown in person detection area ARb of the captured image, it recognizes that no object is shown in object detection area ARc of the captured image, or that even if an object is shown, the object is not a detection target, and proceeds to ACT 23. In ACT 23, processor 51 starts a timer. The timer counts the allowable time before an alarm is issued that the customer has left without paying. The allowable time is several seconds, for example, 5 seconds.

[0056] In ACT24, the processor 51 waits for the timer to time out. When the timer times out, the processor 51 proceeds to ACT25. In ACT25, the processor 51 outputs a notification signal that the attendant has left without making a payment. The notification signal is output to the attendant terminal 20. The notification signal may also be output to the self-service POS terminal 10 or a communication terminal such as a smartphone carried by the store clerk. Incidentally, upon receiving the notification signal, the attendant terminal 20 notifies the attendant AT that the attendant has left without making a payment.

[0057] After outputting the notification signal that the customer has left without paying, the processor 51 proceeds to ACT 26. In ACT 26, the processor 51 clears the person image stored in the second image memory. After that, the processor 51 returns to ACT 1.

[0058] On the other hand, if the item shown in the captured image in ACT 22 is not an item that is not subject to detection, the processor 51 proceeds to ACT 31 in FIG. 9. In ACT 31, the processor 51 stores a leaving image in the third image memory. The leaving image is an image captured at the point in time when the customer who was in the person detection area ARb in ACT 12 is no longer displayed. In the leaving image, no person is displayed in the person detection area ARb. However, items such as a shopping basket, a reusable shopping bag, a plastic shopping bag, and the customer's personal belongings are displayed in the item detection area ARc. The third image memory is part of the volatile memory area in the main memory 52.

[0059] After storing the leaving image, the processor 51 proceeds to ACT 32. In ACT 32, the processor 51 outputs an operation prohibition signal to the self-checkout POS terminal 10. By outputting this operation prohibition signal, the operation device of the self-checkout POS terminal 10 is put into an operation prohibition state.

[0060] After outputting the operation prohibition signal, the processor 51 proceeds to ACT 33. In ACT 33, the processor 51 acquires a captured image. Then, in ACT 34, the processor 51 compares the captured image with the leaving image. In this comparison, the processor 51 checks whether a person is captured in the person detection area ARb of the captured image in ACT 35. If a person is captured in the captured image, or if a person is not captured in the captured image, that is, if there is no difference between the image of the person detection area ARb in the leaving image and the captured image, the processor 51 returns to ACT 33. The processor 51 executes the processes from ACT 32 onwards in the same manner as described above. That is, the processor 51 repeatedly executes the processes from ACT 33 to ACT 35 until a person is captured in the captured image.

[0061] If a person is detected from the captured image in ACT 35, the processor 51 proceeds to ACT 36. In ACT 36, the processor 51 compares the captured image with the person image. Then, in ACT 37, the processor 51 determines whether the person shown in the captured image is the same person as the person shown in the person image. For example, the processor 51 calculates the similarity between the facial image of the person shown in the captured image and the facial image of the person shown in the person image. If the similarity is equal to or greater than a predetermined threshold, the processor 51 determines that the people are the same person. The processor 51 may determine whether the people are the same person based on not only the facial image but also the similarity of images of clothing, footwear, etc.

[0062] If it is determined in ACT 37 that the person is not the same person, the processor 51 returns to ACT 33. The processor 51 executes the processes from ACT 33 onwards in the same manner as described above. In this way, if a person is detected from the captured image but the person is a different person from the person photographed in the person image, the processor 51 repeatedly executes the processes from ACT 33 onwards.

[0063] On the other hand, if it is determined that the person shown in the captured image is the same person as the person shown in the person image, the processor 51 proceeds to ACT 38. The processor 51 stops outputting the operation prohibition signal in ACT 38. In addition, the processor 51 clears the leaving image stored in the third image memory in ACT 39. Thereafter, the processor 51 proceeds to ACT 10 in FIG. 7. The processor 51 executes the processes from ACT 10 onwards in the same manner as described above.

[0064] Here, the processor 51 realizes the function of image storage means 511 by processing ACT1. The processor 51 realizes the function of image comparison means 512 by processing ACT2 and ACT3, ACT6 and ACT7, and ACT10 and ACT11. The processor 51 realizes the function of purchaser detection means 513 by processing ACT4, ACT8 and ACT12. The processor 51 realizes the function of item detection means 514 by processing ACT21. The processor 51 realizes the function of status recognition means 515 by processing ACT5, ACT13 and ACT14. The processor 51 realizes the function of prohibition means 516 by processing ACT31 to ACT38. The processor 51 realizes the function of determination means 517 by processing ACT22. The processor 51 realizes the function of control means 518 by processing ACT21 to ACT26.

[0065] [Effects of monitoring devices] As is clear from the above description, when a purchaser SH starts a transaction-related operation at the self-service POS terminal 10 and then leaves the self-service POS terminal 10 before completing the transaction, the monitoring device 50 of this embodiment checks whether any items have been left in the bagging space SP of the self-service POS terminal 10, on the shopping basket SB mounting table 12, or on the top surface of the automatic change dispenser 13. If no items have been left, the monitoring device 50 outputs a signal to the attendant terminal 20 or the like to notify the purchaser that the purchaser has left without making a transaction. Even if an item has been left, if that item is an item that is not subject to detection, such as cutlery or a receipt, the monitoring device 50 outputs a signal to notify the purchaser that the purchaser has left without making a transaction. In contrast, if an item other than an item that is not subject to detection is left at the self-service POS terminal 10, the monitoring device 50 does not output a signal to notify the purchaser that the purchaser has left without making a transaction.

[0066] For example, a customer SH may temporarily leave the self-service POS terminal 10 during checkout to retrieve a shopping bag or to retrieve an item that they forgot to buy from the sales floor. In such cases, a shopping basket is typically placed on the platform 12. Alternatively, items that have not yet been registered are left there. A personal shopping bag or a shopping bag is also placed in the space SP. In this way, when an item related to the customer SH is placed at the self-service POS terminal 10, the monitoring device 50 does not output a notification signal indicating that the customer has left without checking out.

[0067] In contrast, if the customer SH attempts to leave the checkout area RE without making a payment, no items related to the customer SH will be placed on the self-service POS terminal 10. Even if an item is placed there, it will be an item such as cutlery that the customer SH has determined not to need. In such a case, the monitoring device 50 outputs a notification signal indicating that the customer has left without making a payment.

[0068] In this way, the fraudulent act of leaving without paying is reported when the purchaser SH attempts to leave the checkout area RE without paying. If the purchaser SH goes to get a shopping bag or to the sales floor to get an item they forgot to buy during the checkout process, the fraudulent act is not reported. Therefore, the monitoring device 50 of this embodiment can reduce the risk of false positive reports. Therefore, it is possible to provide a monitoring device 50 that can efficiently and effectively prevent or deter the fraudulent act of leaving without paying.

[0069] Furthermore, the monitoring device 50 stops monitoring when the self-service POS terminal 10 is calling a store clerk. Therefore, even if the customer SH temporarily leaves the store to call a store clerk, the customer will never be notified that they have left without making a payment.

[0070] Furthermore, if the purchaser SH temporarily leaves the self-service POS terminal 10, the monitoring device 50 places the self-service POS terminal 10 in an operation-prohibited state. This operation-prohibited state continues until the purchaser SH returns to the self-service POS terminal 10. Therefore, even if the self-service POS terminal 10 is mistakenly recognized as vacant and another purchaser stands in front of the self-service POS terminal 10, the operation-prohibited state of the self-service POS terminal 10 is not released, so there is no concern that another purchaser will operate the self-service POS terminal 10.

[0071] The monitoring device 50 stores an image of the self-checkout POS terminal 10 taken by the camera 60 from above when the customer SH is not present in the person detection area ARb as an initial image. The monitoring device 50 then compares the image taken by the camera 60 with the initial image. As a result, if a person is captured in the person detection area ARb of the captured image, the monitoring device 50 recognizes that the customer SH is present in front of the self-checkout POS terminal 10. If the images of the customer detection area ARb in the captured image and the initial image are the same, the monitoring device 50 recognizes that the customer SH has left the self-checkout POS terminal 10. In this way, the monitoring device 50 can recognize the event that the customer SH is present in front of the self-checkout POS terminal 10 or that the customer has left the self-checkout POS terminal 10 by comparing the images, which reduces the processing load on the processor 51 and is very efficient compared to when analyzing images using AI.

[0072] The self-service POS terminal to which the monitoring device 50 is applied is not limited to the self-service POS terminal 10 described in this embodiment. For example, it may be a self-service POS terminal that can park a shopping cart instead of the stand 12. Alternatively, it may be a self-service POS terminal that has an automatic change dispenser 13 built into the terminal housing 11 and a space SP for bagging provided on one side of the terminal housing 11. There are also checkout systems in which a purchaser registers products themselves using an information terminal such as a tablet terminal or smartphone, and then pays for the items themselves using a dedicated checkout machine. The monitoring device 50 of this embodiment may be applied to the checkout machine of such a checkout system.

[0073] [Other embodiments] Although one embodiment has been described above, the embodiment is not limited to this. For example, in the above embodiment, one monitoring device 50 is provided for one self-service POS terminal 10. In another embodiment, as in the checkout system 200 shown in FIG. 10 , one monitoring device 500 may be provided for multiple self-service POS terminals 10, and the monitoring device 500 may centrally monitor customers at each self-service POS terminal 10. In this case, the monitoring device 500 captures images captured by each camera 60 provided for each self-service POS terminal 10. The monitoring device 500 also obtains status information indicating the state of each self-service POS terminal 10 via the attendant terminal 20. The monitoring device 500 then uses the status information and images captured by the camera 60 for each self-service POS terminal 10 to detect fraudulent behavior such as leaving without paying.

[0074] For example, in the above embodiment, the front sides of the terminal housing 11, the table 12, and the automatic change dispenser 13 of the self-service POS terminal 10 are defined as the person detection area ARb. In another embodiment, as shown in FIG. 11 , the front side of the terminal housing 11, which is inside the person detection area ARb, is defined as the customer operation area ARd. The monitoring device 50 controls the device so that if a customer SH who was in the customer operation area ARd is no longer detected during a transaction and no items are detected by the item detection means 514, an alert is issued that the customer has left without paying. This ensures that, for example, even if there are multiple people, such as a family, in front of an adjacent self-service POS terminal 10 and one of them enters the person detection area ARb of the self-service POS terminal 10, fraudulent leaving without paying at the self-service POS terminal 10 can be reliably detected.

[0075] In the above embodiment, the person detection area ARb and the article detection area ARc are described as different areas. In this regard, as shown in Fig. 12, the person detection area ARb and the article detection area ARc may be the same area. In other words, the monitoring device 50 may detect whether or not a person is present in the same area, and whether or not an article is present.

[0076] In the above embodiment, the item detection means 514 sets the item detection area ARc not only to the bagging space SP in the self-service POS terminal 10, but also to the top of the mounting table 12 and the automatic change dispenser 13. In another embodiment, the item detection means 514 sets the item detection area ARc only to the bagging space SP. By doing so, the monitoring device 50 can detect fraudulent behavior such as leaving an empty shopping basket SB on the mounting table 12 and leaving without paying.

[0077] For example, in the above embodiment, the purchaser detection means 513 detects the purchaser based on the image comparison result by the first comparison means 5121. In this regard, for example, a human presence sensor or weight sensor may be provided on the front side of the terminal housing 11, and the monitoring device 50 may detect the purchaser based on the detection signals of these sensors.

[0078] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. The inventions described in the original claims of this application are set forth below. [1] A monitoring device comprising: a purchaser detection means for detecting a purchaser at an operating position of a cash register where a transaction-related operation is performed; an item detection means for detecting an item placed on the cash register by the purchaser; and a control means for controlling the device so that, if the purchaser who was detected by the purchaser detection means is no longer detected and the item is no longer detected by the item detection means between the start and end of the transaction-related operation by the purchaser on the cash register, an alert is issued that the purchaser has left without completing the transaction. [2] The monitoring device described in Appendix [1] further comprises a prohibition means for prohibiting operation of the cash register until the purchaser is detected again by the purchaser detection means if the purchaser who was detected by the purchaser detection means is no longer detected and the item is detected by the item detection means between the start and end of the purchaser's operation related to the transaction at the cash register. [3] The monitoring device described in Appendix [2] further comprises a determination means for determining whether the item is an item that is not subject to leaving detection when the purchaser who was detected by the purchaser detection means is no longer detected and the item is detected by the item detection means between the start and end of the purchaser's operation related to the transaction at the cash register, wherein the prohibition means prohibits operation of the cash register if the item is not an item that is not subject to leaving detection, and the control means controls so that leaving without paying is notified if the item is an item that is not subject to leaving detection. [4] The monitoring device described in Appendix [1] further comprises an image storage means for storing a camera image taken from above of the cash register when the customer is not present at the operating position, and an image comparison means for comparing the image taken by the camera after the cash register operation has begun with the image stored by the image storage means, wherein the customer detection means and the item detection means detect the customer and the item based on the image comparison results by the image comparison means. [5] The image comparison means includes a first comparison means that compares the captured image with an image stored by the image storage means for a person detection area including the operating position of the cash register, and a second comparison means that compares the captured image with an image stored by the image storage means for an item detection area including the location where the item is placed on the cash register, and the customer detection means detects the customer based on the image comparison result by the first comparison means, and the item detection means detects the item based on the image comparison result by the second comparison means, in the monitoring device described in Appendix [4]. [6] A program for causing a computer of a monitoring device that monitors an accounting machine where operations related to the payment of purchased goods are performed to function as a purchaser detection means for detecting a purchaser at the operating position of the accounting machine, an item detection means for detecting items placed on the accounting machine by the purchaser, and a control means for controlling the computer so that, if the purchaser who was detected by the purchaser detection means is no longer detected and the item is not detected by the item detection means between the start and end of the purchaser's operations related to the payment on the accounting machine, an alert is issued that the purchaser has left without paying. [Explanation of symbols]

[0079] 10...self-service POS terminal, 11...terminal housing, 12...mounting stand, 13...automatic change dispenser, 14...touch panel, 15...scanner, 16...receipt printer, 20...attendant terminal, 30...store server, 40...communication network, 50...monitoring device, 51...processor, 52...main memory, 53...auxiliary storage device, 54...clock, 55...terminal interface, 56...camera interface, 57...network interface, 58...system transmission path, 60...camera, 100...checkout system, 200...checkout system, 500...monitoring device, 511...image storage means, 512...image comparison means, 513...purchaser detection means, 514...item detection means, 515...status recognition means, 516...prohibition means, 517...determination means, 518...control means, 5121...first comparison means, 5122...second comparison means.

Claims

1. a customer detection means for detecting a customer who is present at an operating position of a checkout machine where operations related to the transaction are performed; an item detection means for detecting an item placed on the cash register by the purchaser; a prohibition means for prohibiting operation of the checkout machine until the purchaser is detected again by the purchaser detection means if the purchaser who was detected by the purchaser detection means is no longer detected and the item is detected by the item detection means between the start and end of the operation related to the checkout by the purchaser at the checkout machine; A monitoring device comprising:

2. a determination means for determining whether the item is an item that is not subject to leaving detection when the purchaser who was detected by the purchaser detection means is no longer detected and the item is detected by the item detection means between the start and end of the purchaser's operation related to the transaction at the checkout machine; Further comprising:

2. The monitoring device according to claim 1, wherein said prohibiting means prohibits operation of said cash register when said article is not an article that is not subject to leaving detection.

3. a control means for controlling the device so that, if the purchaser who had been detected by the purchaser detection means is no longer detected and the item is not detected by the item detection means during the period from the start to the end of the operation related to the transaction by the purchaser on the transaction machine, a warning is given that the purchaser has left without making a transaction; 10. The monitoring device of claim 1, further comprising:

4. an image storage means for storing an image taken by a camera from above the checkout machine when the customer is not present at the operating position; an image comparison means for comparing the image captured by the camera after the transaction operation has begun with the image stored by the image storage means; Further comprising:

2. The monitoring device according to claim 1, wherein said purchaser detection means and said item detection means detect said purchaser and said item based on the result of comparison of images by said image comparison means.

5. The image comparison means a first comparison means for comparing the captured image with an image stored by the image storage means for a person detection area including the operation position of the cashier; a second comparison means for comparing the captured image with an image stored by the image storage means for an item detection area of ​​the checkout machine that includes a location where the item is placed; Including, the purchaser detection means detects the purchaser based on a comparison result of the image by the first comparison means, 5. The monitoring device according to claim 4, wherein said object detecting means detects said object based on a result of comparison of images by said second comparing means.

6. a computer of a monitoring device that monitors an accounting machine where operations related to accounting for purchased goods are performed; a customer detection means for detecting a customer at the operating position of the cashier; an item detection means for detecting an item placed on the cash register by the purchaser; and a prohibition means for prohibiting operation of the checkout machine until the purchaser is detected again by the purchaser detection means, if the purchaser who was detected by the purchaser detection means becomes undetectable and the item is detected by the item detection means between the start and end of the operation related to the checkout by the purchaser; A program to function as a

Images