Monitoring system, monitoring device, monitoring method, and program

JPWO2023170784A5Active Publication Date: 2024-11-13NEC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024505698
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-03-08
Filing Date
2022-03-08
Publication Date
2024-11-13
Estimated Expiration
2042-03-08

AI Technical Summary

Technical Problem

Self-checkout systems face challenges in preventing accounting omissions for products around payment devices operated by customers, as customers may inadvertently or intentionally fail to register items during the product registration and payment process.

Method used

A monitoring system that includes an image acquisition unit to capture images of the product display area, a detection unit to identify when a customer takes a product, and a processing unit to prompt registration if the accounting process is incomplete, with information stored for customer identification after completion.

Benefits of technology

Effectively reduces accounting omissions by prompting customers to register products during the checkout process and ensuring accurate recording of items purchased, enhancing the security and reliability of self-checkout transactions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

A monitoring device (100) comprises: an image acquisition unit (102) that acquires an image including a product display area within a predetermined range from a payment device that is operated by a customer and has a product registration function and payment function; a detection unit (104) that processes the image to detect that a product has been taken from the product display area by the customer; and a processing unit (106) that, when the detection unit (104) has detected that a product has been taken by the customer, causes an output device to output information for prompting processing of registering the product if payment processing is incomplete, or stores information enabling identification of the customer into a storage device if payment processing is complete.
Need to check novelty before this filing date? Find Prior Art

Description

Monitoring system, monitoring device, monitoring method, and recording medium

[0001] The present invention relates to a monitoring system, a monitoring device, a monitoring method, and a recording medium.

[0002] In recent years, retail stores and other businesses have been forced to reduce the number of employees in order to alleviate labor shortages and reduce labor costs. Therefore, so-called self-checkout registers, which allow customers to register products and process payments themselves, have become widespread. However, because self-checkout registers are operated by customers themselves, there is a risk of fraud by customers, and therefore countermeasures against fraud are necessary.

[0003] An example of a checkout system that takes measures against such fraudulent activities is described in Patent Document 1. This checkout system includes a clerk registration means through which a clerk registers items; a checkout means operated by a customer to perform checkout; an imaging means for capturing images of the customer's operations during checkout; a display means for displaying the captured images; a state control means for controlling the system between a first state in which the clerk registration means primarily registers items and a second state; and a display control means for changing the display control of the display means depending on the state controlled by the state control means. With this configuration, the checkout system described in Patent Document 1 can capture a series of registration actions, such as removing items from a shopping cart, registering items, and adding registered items to the shopping cart, on a wide-area image capture screen. This makes customers aware of the monitoring in full self-service mode, thereby effectively deterring fraudulent activities related to checkout.

[0004] Furthermore, Patent Documents 2 to 4 describe systems that automatically register products by identifying products taken from shelves and adding them to the customer's purchase list. In each of these documents, images captured by a camera or data detected by a weight sensor are used to detect that a product has been taken from a shelf based on the customer's position and movement, and the customer's movements within the store are tracked, thereby identifying and linking the product taken from the shelf with the customer who took it, and enabling automatic payment.

[0005] Japanese Patent Application Publication No. 2021-157420 Japanese Patent Application Publication No. 2021-108208 Japanese Patent Application Publication No. 2020-115261 International Publication No. 2021 / 125357

[0006] The technologies described in Patent Documents 2 to 4 relate to technologies for automatically registering products and performing settlement. Therefore, the technologies described in Patent Documents 2 to 4 do not require customers to register products or perform settlement processes, so there is no risk of missing items or customers committing fraud during product registration or settlement processes. Furthermore, the technology described in Patent Document 1 prevents fraudulent acts during settlement, such as falsely or mistakenly reporting an amount higher than the amount inserted. Therefore, the technology described in Patent Document 1 cannot prevent missing items from being registered.

[0007] In response to this, the inventor focused on the problem that after a customer picks up a product near a payment device that is operated by the customer and has product registration and payment functions, the product may frequently be omitted from the payment, either intentionally or inadvertently.

[0008] In view of the above-mentioned problems, one example of the object of the present invention is to provide a monitoring system, a monitoring device, a monitoring method, and a recording medium that solve the problem of missing a certain product in the vicinity of a checkout device that is operated by a customer and has product registration and checkout functions.

[0009] According to one aspect of the present invention, a monitoring device is provided that includes: an image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device that has product registration and checkout functions; a detection means that detects that the customer has taken a product from the product display area by processing the image; and a processing means that, when the detection means detects that the customer has taken the product, outputs information to an output means that prompts the customer to perform registration processing for the product if the checkout process has not yet been completed, or stores information that can identify the customer in a storage means if the checkout process has been completed.

[0010] According to one aspect of the present invention, a monitoring system is provided, comprising: a monitoring device; a checkout device that is operated by a customer and has product registration and checkout functions; and a camera that generates an image including a product display area within a predetermined range of the checkout device, wherein the monitoring device has an image acquisition means that acquires the image; a detection means that processes the image to detect that a customer has taken a product from the product display area; and a processing means that, when the detection means detects that the customer has taken the product, outputs information to an output means that prompts the customer to register the product if the checkout process has not yet been completed, or stores information that can identify the customer in a storage means if the checkout process has been completed.

[0011] According to one aspect of the present invention, a monitoring method is provided in which one or more computers acquire an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions, process the image to detect that the customer has taken a product from the product display area, and when it is detected that the customer has taken the product, if the checkout process has not yet been completed, output information to an output means to prompt the customer to perform a registration process for the product, or if the checkout process has been completed, store information that can identify the customer in a storage means.

[0012] According to one aspect of the present invention, a computer-readable recording medium is provided having recorded thereon a program for causing a computer to execute the following steps: acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions; detecting that the customer has taken a product from the product display area by processing the image; and, when it is detected in the detection step that the customer has taken the product, outputting information from an output means that prompts the customer to perform registration processing for the product if the checkout process has not yet been completed; and storing information that can identify the customer in a storage means if the checkout process has been completed.

[0013] Another aspect of the present invention may be a program that causes at least one computer to execute the method of the above aspect, or a computer-readable recording medium having such a program recorded thereon. This recording medium includes a non-transitory tangible medium. This computer program includes computer program code that, when executed by a computer, causes the computer to implement the monitoring method on a monitoring device.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention.

[0015] Furthermore, the various components of the present invention do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be part of another component, or part of one component may overlap with part of another component, etc.

[0016] Furthermore, although the method and computer program of the present invention describe a number of steps in a sequential order, the order in which the steps are described does not limit the order in which the steps are executed. Therefore, when implementing the method and computer program of the present invention, the order of the steps can be changed as long as it does not cause any problems in terms of the content.

[0017] Furthermore, the multiple steps of the method and computer program of the present invention are not limited to being executed at different times, and therefore, a step may occur while another step is being executed, or the execution timing of a step may partially or completely overlap with the execution timing of another step, etc.

[0018] According to one aspect of the present invention, a monitoring system, a monitoring device, a monitoring method, and a recording medium are provided that solve the problem of missing an item in the vicinity of a checkout device that is operated by a customer and has product registration and checkout functions.

[0019] 1 is a diagram illustrating an overview of a monitoring device according to an embodiment. FIG. 1 is a flowchart illustrating an example of the operation of the monitoring device according to an embodiment. FIG. 2 is a diagram illustrating an example of an image acquired by the monitoring device. FIG. 3 is a diagram conceptually illustrating a system configuration of a monitoring system according to an embodiment. FIG. 4 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the monitoring device. FIG. 5 is a diagram illustrating an example of the data structure of a detection result according to an embodiment. FIG. 6 is a functional block diagram illustrating an example of the logical configuration of the monitoring device according to an embodiment. FIG. 7 is a diagram illustrating an example of the data structure of a detection result according to an embodiment. FIG. 8 is a flowchart illustrating an example of the operation of the monitoring device according to an embodiment. FIG. 9 is a functional block diagram illustrating an example of the logical configuration of the monitoring device according to an embodiment. FIG. 10 is a flowchart illustrating an example of the operation of the monitoring device according to an embodiment. FIG. 11 is a diagram illustrating an example of the data structure of a detection result in which a determination flag is stored. FIG. 12 is a functional block diagram illustrating an example of the logical configuration of the monitoring device according to an embodiment. FIG. 13 is a flowchart illustrating an example of the operation of the monitoring device according to an embodiment. Fig. 10 is a diagram for explaining the output timing of a message. Fig. 11 is a diagram showing an operation example of Example 3. Fig. 12 is a diagram showing an operation example of Example 4. Fig. 13 is a diagram showing an operation example of Example 5. Fig. 14 is a diagram showing an operation example of Example 6. Fig. 15 is a diagram showing an operation example of Example 7. Fig. 16 is a diagram showing an operation example of Example 8.

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by similar reference numerals, and their description will be omitted as appropriate. In addition, in the following drawings, configurations of parts that are not related to the essence of the present invention are omitted and are not shown.

[0021] In the embodiments, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition); and inputting data or information output from another device into the device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a response, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0022] <Minimum Configuration Example> FIG. 1 is a diagram illustrating an overview of a monitoring device 100 according to an embodiment. The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device having product registration and checkout functions. The detection unit 104 processes the image to detect that the customer has taken a product from the product display area. When the detection unit 104 detects that the customer has taken a product, the processing unit 106 performs the following processes (a) and (b). In process (a), if the transaction process has not yet been completed, the output device is caused to output information prompting the customer to perform registration processing for the product. In process (b), if the transaction process has already been completed, the storage device is caused to store information capable of identifying the customer.

[0023] <Operation Example> Fig. 2 is a flowchart showing an operation example of the monitoring device 100 according to the embodiment. Fig. 3 is a diagram showing an example of an image captured of a customer making an impulse purchase of a product during product registration processing at a checkout device operated by the customer. The following description will be given with reference to Figs. 2 and 3.

[0024] First, the image acquisition unit 102 acquires an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration function and a payment settlement function (step S101). The detection unit 104 processes the image to detect that the customer has taken a product from the product display area (step S103).

[0025] As shown in Fig. 3(a), promotional products 10 are displayed in a product display area 12 around the checkout device (self-checkout terminal 50) operated by the customer, to encourage customers who are currently registering products to make impulse purchases. The detection unit 104 then uses image processing to detect that the customer's hand 60 has entered the product display area in the image, as shown in Fig. 3(b), thereby detecting that the customer may have taken a product or that the customer's hand 60 is holding a product.

[0026] When the detection unit 104 detects that a customer has picked up a product (YES in step S103), the processing unit 106 performs the following processes (a) and (b). In process (a), if the transaction has not yet been completed (NO in step S105), the processing unit 106 causes the output device to output information prompting the customer to register the product (step S109). In process (b), if the transaction has already been completed (YES in step S105), the processing unit 106 stores information that can identify the customer in the storage device (step S107).

[0027] In the monitoring device 100, the image acquisition unit 102 acquires an image including a product display area within a predetermined range from a checkout device operated by a customer, and the detection unit 104 processes the image to detect that the customer has taken a product from the product display area. When the detection unit 104 detects that the customer has taken a product, the processing unit 106 performs the following processes (a) and (b). Process (a), if the transaction has not yet been completed, causes the output device to output information prompting the customer to register the product. Process (b), if the transaction has already been completed, causes the storage device to store information that can identify the customer.

[0028] According to this monitoring device 100, a monitoring system, monitoring device, monitoring method, and recording medium can be obtained that are operated by a customer and solve the problem of missing an item in the accounting process around a checkout device that has product registration and checkout functions.

[0029] A detailed example of the monitoring device 100 will be described below.

[0030] (First embodiment) <System overview> Fig. 4 is a diagram conceptually illustrating the system configuration of a monitoring system 1 according to an embodiment. The monitoring system 1 includes a monitoring device 100, at least one camera 5, and at least one self-checkout terminal 50, which are connected to the monitoring device 100 via a communication network 3. The monitoring device 100 includes a storage device 120. The storage device 120 may be provided inside the monitoring device 100 or may be provided externally. In other words, the storage device 120 may be hardware that is integrated with the monitoring device 100, or may be hardware that is separate from the monitoring device 100.

[0031] The self-checkout terminal 50 is a payment device operated by a customer and has product registration and payment functions. The self-checkout terminal 50 has at least a touch panel 52 and a scanner 54. The self-checkout terminal 50 may also include a printer that prints out receipts and the like. The touch panel 52 includes a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and the display unit and operation unit are integrated together.

[0032] The scanner 54 reads barcodes containing product information to register products purchased by customers. The scanner 54 may be a code reader equipped with a light source and a light-receiving element such as a photodiode to detect product codes for identifying products, or a camera equipped with a lens and an imaging element such as a CCD (Charge Coupled Device) image sensor to capture images including the product's exterior. The scanner 54 may be either a handheld type that the customer holds in their hand or a stationary type that reads product codes by bringing the scanner close to the reading surface.

[0033] The self-checkout terminal 50 may further include a magnetic card reader or an IC card reader that reads store membership cards, credit cards, and cash cards (magnetic cards or IC cards) from financial institutions, etc. Alternatively, the self-checkout terminal 50 may include an IC card reader that reads transportation IC cards and IC cards from mobile devices such as smartphones via contactless IC communication (NFC (Near Field Communication) communication). Furthermore, the self-checkout terminal 50 may include a reader that reads QR (Quick Response) Code (registered trademark) QR codes for payment. At least one of these readers and the scanner 54 may be integrated.

[0034] The camera 5 includes an imaging element such as a lens and a charge coupled device (CCD) image sensor, and is, for example, a network camera such as an Internet Protocol (IP) camera. The network camera has, for example, a wireless local area network (LAN) communication function and is connected to the monitoring device 100 via a communication network 3, i.e., a relay device (not shown) such as a router. These cameras 5 may be so-called monitoring cameras, with multiple cameras installed in a store or facility. The camera 5 may also include a mechanism for controlling the orientation of the camera body and lens, zooming, focusing, etc., in accordance with the movement of a person.

[0035] The images generated by the camera 5 are preferably taken in real time and transmitted to the monitoring device 100. However, the images transmitted to the monitoring device 100 do not have to be transmitted directly from the camera 5, but may be images delayed by a predetermined time. The images captured by the camera 5 may be temporarily stored in another storage device, and the monitoring device 100 may read them out from the storage device sequentially or at predetermined intervals. Furthermore, the images transmitted to the monitoring device 100 are preferably moving images, but may also be frame images taken at predetermined intervals or still images.

[0036] The monitoring device 100 is further connected to an image processing device 130. However, the image processing device 130 may be provided inside or outside the monitoring device 100. In other words, the image processing device 130 may be hardware that is integrated with the monitoring device 100, or may be hardware that is separate from the monitoring device 100.

[0037] The image processing device 130 performs image recognition processing on the image captured by the camera 5 to identify the positions of the product display area 12, the product 10, and the customer's hand 60 in the image. Furthermore, the image processing device 130 may track the movement of at least one of the product 10 and the customer's hand 60. Furthermore, the image processing device 130 may recognize that the customer's hand 60 has picked up or held the product 10. Furthermore, the image processing device 130 may register features of each product 10 (including plastic bags, cups for self-service drinks, and the following various items in the case where a fee is charged, such as chopsticks, forks, spoons, and straws) in advance, and identify the product 10 in the image by processing a comparison with the features of the product 10.

[0038] <Hardware Configuration Example> Fig. 5 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the monitoring device 100. The self-checkout terminal 50 and image processing device 130 in Fig. 4 are also realized by the computer 1000. Furthermore, the camera 5 in Fig. 4 may also be realized by the computer 1000. Furthermore, the functions of the monitoring device 100 may be shared and realized by at least two of the self-checkout terminal 50, the monitoring device 100, and the camera 5.

[0039] The computer 1000 has a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0040] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0041] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0042] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0043] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the monitoring device 100 (e.g., the image acquisition unit 102, the detection unit 104, the processing unit 106, the determination unit 108 described below, the identification unit 110, the registration unit 112, etc.). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also store each data of the storage device 120 of the monitoring device 100.

[0044] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded may include a non-transitory, tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0045] The input / output interface 1050 is an interface for connecting the computer 1000 with various input / output devices. The input / output interface 1050 also functions as a communication interface for performing short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication).

[0046] The network interface 1060 is an interface for connecting the computer 1000 to a communication network. This communication network may be, for example, a local area network (LAN) or a wide area network (WAN). The network interface 1060 may connect to the communication network wirelessly or by wire.

[0047] The computer 1000 then connects to necessary devices (for example, the touch panel 52 of the self-checkout terminal 50, the scanner 54, a speaker, a printer, various readers, etc.) via the input / output interface 1050 or the network interface 1060.

[0048] The components of the monitoring device 100 in each of the embodiments shown in Figure 1 and Figures 7, 12, and 15 described below are realized by any combination of hardware and software of the computer 1000 shown in Figure 5. Those skilled in the art will understand that there are many variations in the realization methods and devices. The functional block diagrams showing the monitoring device 100 in each embodiment show logical functional blocks rather than a hardware-based configuration.

[0049] <Example of Functional Configuration> An example of the functional configuration of the monitoring device 100 according to the embodiment will be described with reference to Fig. 1. The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 is operated by a customer and acquires an image including the product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50 in Fig. 4) that has a product registration function and a checkout function.

[0050] The predetermined range is, for example, 2 meters or less. Alternatively, the predetermined range may be the range within reach of the hand 60 of a customer operating the self-checkout terminal 50 or the range within the customer's line of sight. In other words, the predetermined range is preferably a distance at which a customer operating the self-checkout terminal 50 can easily notice the presence of the product 10 and pick it up. The product 10 is, for example, a product that is the subject of a sales promotion, and is displayed in the product display area 12 so as to entice the customer operating the self-checkout terminal 50 to make an impulse purchase.

[0051] The products 10 also include plastic bags 20, which are subject to a fee, and coffee cups 30 for use in self-service coffee machines. The products 10 are provided with barcodes for product registration. However, the plastic bags 20 and coffee cups 30 do not necessarily have to have barcodes for product registration; for example, dedicated barcodes (or two-dimensional codes) may be provided near the plastic bags 20 and coffee cups 30. Alternatively, the use of plastic bags 20 and the type of coffee to be served with the coffee cups 30 may be registered on the product registration screen of the self-service checkout terminal 50.

[0052] The detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12. The detection unit 104 causes the image processing device 130 to process the image captured by the camera 5. The image processing device 130 uses image recognition processing to detect that a customer has taken a product 10 from the product display area 12. The detection method will be described in detail in the embodiment described below.

[0053] When the detection unit 104 detects that a customer has taken a product 10, the processing unit 106 performs the following processes (a) and (b). In process (a), if the transaction has not yet been completed, the processing unit 106 causes the output device to output information prompting the customer to perform the registration process for the product 10. In process (b), if the transaction has already been completed, the processing unit 106 stores information that can identify the customer in the storage device 120.

[0054] "Before the transaction is completed" refers to the period from the moment the customer stands in front of the self-checkout terminal 50 and the start transaction button on the standby screen displayed on the touch panel 52 is pressed, including during the product registration process and the payment procedure, until the customer has received their change and receipt. "Before the transaction is completed" may also refer to the period from the moment the customer stands in front of the self-checkout terminal 50 until the moment the customer leaves the self-checkout terminal 50.

[0055] The following are examples of methods for determining whether the transaction has not yet been completed. At least one of the following determination conditions may be used, or at least two of them may be used in combination. (a1) While the image from the camera 5 is processed to detect that a customer is standing in front of the self-checkout terminal 50 (the customer detection process may be performed by the detection unit 104, for example) (However, the presence of the customer may also be detected by various types of human sensors, such as infrared, visible light, piezoelectric, electrostatic, and ultrasonic sensors, but the sensor type is not limited to these.) (a2) After the self-checkout terminal 50 accepts the customer pressing the start transaction button on the standby screen displayed on the touch panel 52 of the self-checkout terminal 50. (a3) ​​While the self-checkout terminal 50 is accepting the customer's product registration operation. (a4) While the self-checkout terminal 50 is accepting the customer's payment operation. (a5) While the self-checkout terminal 50 detects that the receipt and change have not been received by the customer.

[0056] "After the transaction process is completed" refers to the period from when the customer receives their change and receipt at the self-checkout terminal 50 until they leave the store. Alternatively, "after the transaction process is completed" may refer to the period from when the customer leaves the self-checkout terminal 50 until they leave the store.

[0057] The method for determining whether or not the transaction has been completed is exemplified below. At least one of the following determination conditions may be used, or at least two of them may be used in combination. (b1) After the image from camera 5 is processed and it is detected that the customer in front of the self-checkout terminal 50 has left. (b2) After it is detected that the self-checkout terminal 50 has received the receipt or change from the customer. (b3) While the image from camera 5 is processed and it is possible to track the customer within the store, or while it is detected that the customer is present within the store. (b4) After the image from camera 5 is processed and it is detected that the customer has left the self-checkout terminal 50, and before it is detected that the customer has left the store exit.

[0058] The output device may include, for example, at least one of the touch panel 52 of the self-checkout terminal 50, a speaker of the self-checkout terminal 50, and signage or a speaker located near the self-checkout terminal 50, and is not particularly limited as long as the customer can view the output information on the spot. If the output device is a display device, it displays a screen displaying information prompting the customer to register the product. If the output device is a speaker, it outputs a message prompting the customer to register the product by voice. Alternatively, information notifying the customer that there is an unregistered product may be printed on the receipt output from the self-checkout terminal 50. In other words, the output device may be a printer of the self-checkout terminal 50. Furthermore, an electronic receipt including information notifying the customer that there is an unregistered product may be sent to a device such as a customer's smartphone on which an application capable of receiving electronic receipts (hereinafter referred to as an electronic receipt) has been pre-installed. In other words, the output device may include a customer's device such as a smartphone or tablet on which an application capable of receiving and displaying an electronic receipt has been installed.

[0059] For example, when a customer picks up a product with a product name of "CCC Gum," the information to be output may be "Please scan the product (which may be the product name "CCC Gum")," "Did you forget to scan the product (which may be the product name "CCC Gum")?", "There is a product that has not been scanned," "CCC Gum is a product that needs to be scanned," etc. In other words, the information to be output includes at least one of the following: content urging the customer to register the product; content informing the customer that registration processing has not been performed for the product; and content informing the customer that registration processing for the product is required.

[0060] Furthermore, the information to be output may include content emphasizing that the customer has just picked up an item, such as "Please scan the item you just picked up," or "The item you just picked up needs to be scanned." This will make the customer aware that their actions are being monitored, which will likely deter fraudulent behavior.

[0061] Furthermore, as described above, when a customer takes an item including a shopping bag, a cup for self-service drinks (for example, a coffee cup), and the following various items that are subject to a fee, such as chopsticks, forks, spoons, and straws, information can be output in which the product name is replaced with the product name, such as a shopping bag or a coffee cup (or self-service coffee). In the case of these items, the output information may be an operation screen that accepts the selection of use on the touch panel 52 of the self-checkout terminal 50.

[0062] For example, if it is detected that a customer has taken a plastic bag or coffee cup, the processing unit 106 may cause the touch panel 52 of the self-checkout terminal 50 to display an operation screen including information such as "Please enter the number of plastic bags you have taken," "There is a charge for the plastic bags you have taken. Please select the number you would like to use," and "Please select the type of coffee you would like to purchase," as well as a UI (User Interface) that accepts customer operations.

[0063] The information that can identify a customer includes at least one of the following: identification information of the electronic payment medium used by the customer for the transaction, a credit card number, a store membership card number (customer identification information), an image containing the customer, and human features extracted from the image.

[0064] The customer-identifiable information can be, for example, information acquired by the self-checkout terminal 50. Alternatively, the customer-identifiable information can be acquired by using an image captured by the camera 5 or by extracting features by processing the image processing device 130.

[0065] The image including the customer and the image from which the feature amount is extracted may be taken by a camera other than the camera 5, or may be taken by the camera 5.

[0066] <Operation Example> An operation example of the monitoring device 100 will be described with reference to Fig. 2. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). For example, an image acquired by the image acquisition unit 102 is illustrated in Fig. 3(a).

[0067] The detection unit 104 then causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that the customer has taken the product 10 from the product display area 12 (step S103). The image acquisition process of step S101 and the detection process of step S103 may be repeatedly executed while the customer is in front of the self-checkout terminal 50.

[0068] As shown in Fig. 3(a), around the checkout device (self-checkout terminal 50) operated by the customer, promotional products 10 are displayed in a product display area 12 to encourage customers who are undergoing product registration processing to make an impulse purchase. Then, as shown in Fig. 3(b), when a customer's hand 60 enters the product display area 12 and picks up a product 10 in the image, the detection unit 104 detects that the customer has picked up the product 10.

[0069] 6A, the detection unit 104 may record the result of detection of the taking of the product 10 in the detection result 200. The detection result 200 stores information indicating at least one of the following: image data of the product taken, identification information of the image (e.g., frame number), the time of photographing, identification information of the self-checkout terminal 50 operated by the customer, and the name of the image data.

[0070] When the detection unit 104 detects that a customer has picked up a product (YES in step S103), the processing unit 106 performs the following processes (a) and (b). In process (a), if the transaction has not yet been completed (NO in step S105), the processing unit 106 outputs information to the output device prompting the customer to register the product (step S109). In process (b), if the transaction has already been completed (YES in step S105), the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0071] In step S109, the processing unit 106 may display information such as "Please scan the product" on the touch panel 52 of the self-checkout terminal 50 or output the information as audio from the speaker of the self-checkout terminal 50. Then, the process may return to step S101 and continue monitoring whether the customer picks up another product 10.

[0072] In step S107, the processing unit 106 may add the customer-identifying information to the detection result 200 and record it in the storage device 120, as shown in FIG. 6B. Then, this process ends. The process may return to step S101 to prepare for the next customer.

[0073] As described above, according to this embodiment, the monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 operated by the customer, and the detection unit 104 processes the image to detect that the customer has taken a product 10 from the product display area 12. When the detection unit 104 detects that the customer has taken a product 10, the processing unit 106 performs the following processes (a) and (b). In process (a), if the transaction has not yet been completed, the output device is caused to output information prompting the customer to register the product. In process (b), if the transaction has already been completed, the storage device 120 is caused to store information that can identify the customer.

[0074] Specifically, when the detection unit 104 detects that a customer has picked up an item 10 located near the self-checkout terminal 50, if the transaction process at the self-checkout terminal 50 has not yet been completed, the processing unit 106 can display information on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the picked up item 10, or output audio from the speaker of the self-checkout terminal 50. Alternatively, after the detection unit 104 detects that a customer has picked up an item 10 located near the self-checkout terminal 50, the transaction process at the self-checkout terminal 50 is completed and the customer leaves, information that can identify the customer can be recorded in the detection result 200, allowing the store to manage the customer. This makes it possible to take measures to prevent transaction omissions.

[0075] As described above, the monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0076] Second Embodiment Fig. 7 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to the above-described embodiment, except that it has a configuration for determining whether the product picked up by a customer is included in the list of products read by the self-checkout terminal 50. The monitoring device 100 according to this embodiment further includes a determination unit 108 in addition to the configuration of the monitoring device 100 of Fig. 1. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 of Fig. 1, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0077] <Example of functional configuration> Before the transaction process is completed, the determination unit 108 determines whether the product 10 picked up by the customer is included in the product list read by the self-checkout terminal 50 during the subsequent product registration process. If the processing unit 106 determines that the product 10 is not included in the list, it causes the output device to output information prompting the customer to continue the product registration process.

[0078] The list of products read by the self-checkout terminal 50 is a list of POS data that includes at least the product names and prices of the products read by the self-checkout terminal 50 during the product registration process. Alternatively, the POS data may include the product codes of the products.

[0079] The determination unit 108 refers to the list of POS data and determines whether the list includes the product 10 detected by the detection unit 104 as having been taken by the customer. For example, the determination result may be stored in association with the detection result 200. The determination result includes at least one of information indicating whether the list included the POS data for the product 10, and information indicating that the transaction is legitimate if the POS data is present or that the transaction is fraudulent if the POS data is absent.

[0080] The method for identifying the product 10 picked up by a customer is exemplified below. The information on the product 10 identified here is, for example, at least one of the product name and product code of the product 10. (c1) The feature quantities of the product 10 are registered in advance, and the image processing device 130 performs image recognition processing on an image of the product 10 to identify the product. (c2) Products displayed in each product display area 12 are registered in advance. By identifying the product display area 12 in the image, the product associated with the product display area 12 based on the position of the customer's hand 60 is identified as the product 10 picked up by the customer.

[0081] Fig. 8 is a diagram showing an example of the data structure of the detection result 200. In this example, the detection result 200 in Fig. 6 is further linked with information indicating the presence or absence of POS data and information indicating whether the data is fraudulent or legitimate.

[0082] The "subsequent product registration process" refers to the product registration process performed at the self-checkout terminal 50 after it is detected that the customer has taken the product 10.

[0083] Here, the output device may be the same as that in the first embodiment. The information prompting the product registration process may be the same as the information output after it is detected that the customer has taken the product 10, or may include information that further urges the customer to proceed with the registration process. That is, when it is detected that the customer has taken the product 10, the processing unit 106 outputs information that initially calls for attention to prompt the customer to proceed with the registration process, and then, if it is confirmed that the product 10 has not been scanned, outputs information to further emphasize the need to proceed with the registration process.

[0084] For example, if a customer picks up a product with the product name "CCC Gum," the information to be output may be, "There is a product that has not yet been scanned. Please scan the product (which may have the product name "CCC Gum")," "It seems that you forgot to scan the product (which may have the product name "CCC Gum"). Please scan it," "CCC Gum is chargeable. Please scan it," etc.

[0085] That is, the information to be output includes at least one of the following: a content urging the customer to re-register the product; a content informing the customer that they have forgotten to register the product; and a content informing the customer that the product is subject to a fee.

[0086] In another example, the processing unit 106 outputs an alert to an output device when it is determined that the product 10 is not included in the list. This process by the processing unit 106 is also referred to as process (c). In this example, the output device includes at least one of the touch panel 52 of the self-checkout terminal 50, a speaker, signage, a monitoring device in the monitoring center, a manager terminal, and a store clerk terminal. For the customer, the processing unit 106 displays a warning message on the touch panel 52 of the self-checkout terminal 50 (for example, by flashing or highlighting an icon) or outputs an alert sound from the speaker to alert the customer.

[0087] The alert may include information with a stronger message than information urging the customer to register the product. For example, the alert preferably includes information informing the customer that the product they have picked up will be charged for and that it needs to be scanned, and instructing the customer to scan it, such as "Please scan the product you have picked up," "The product you have picked up is subject to a fee. Please scan it," "You forgot to scan the product you have picked up. Please scan it," or "The product you have picked up needs to be scanned. Please scan it."

[0088] Furthermore, the processing unit 106 notifies the manager or store clerk by outputting a warning message including information informing the manager or store clerk that there is a customer who has not registered the product 10 to the device or terminal of the manager or store clerk. The processing unit 106 may send the message including information that can identify the customer to the terminal, or may send the message including information that can identify the self-checkout terminal 50 used by the customer to the terminal.

[0089] This configuration makes it possible to strongly encourage customers to register their products. In addition, since it is possible to notify a manager or store clerk, a nearby store clerk can directly call out to the customer and encourage them to register their products.

[0090] The following describes the operation of the monitoring device 100 according to the embodiment. Hereinafter, we will explain an operation example 1 in which, when there is no POS data, information prompting product registration is output again, and an operation example 2 in which an alert is output to the output device.

[0091] <Operation Example 1> Figure 9 is a flowchart showing an operation example of the monitoring device 100 according to an embodiment. This flow is executed after step S109 in Figure 2. After the processing unit 106 outputs information prompting product registration in step S109, the determination unit 108 determines whether the product 10 picked up by the customer is included in the list of products read by the self-checkout terminal 50 during the subsequent product registration process (step S201). Specifically, the POS data list of the self-checkout terminal 50 is referenced to determine whether the POS data of the product 10 picked up by the customer is included in the list.

[0092] If the processing unit 106 determines that the product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output information prompting the user to proceed with product registration (step S203). For example, the processing unit 106 causes the touch panel 52 of the self-checkout terminal 50 to display a message such as, "It seems you forgot to scan the product (which may be "CCC Gum"). Please scan it." Then, the process returns to step S101 in FIG. 2.

[0093] If it is determined that the product 10 is included in the list (YES in step S201), the processing unit 106 ends this process. Alternatively, the processing unit 106 may return to step S101 in FIG. 2 and start monitoring whether another product 10 is taken.

[0094] In this way, if it is detected that the customer has taken the product 10 and the POS data list does not include the POS data of the product 10, information prompting the customer to register the product can be output again.

[0095] <Operation Example 2> Fig. 10 is a flowchart showing an operation example of the monitoring device 100 according to the embodiment. This flow is also executed after step S109 in Fig. 2. Step S201 is the same as the flow in Fig. 9. Then, if it is determined that the product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output an alert (step S221). Then, the process returns to step S101 in Fig. 2.

[0096] Specifically, the processing unit 106 displays a warning message on the touch panel 52 of the self-checkout terminal 50 (for example, by flashing or highlighting an icon) and outputs an alert sound from the speaker to draw attention to the customer. Furthermore, the processing unit 106 notifies the manager or store clerk by outputting a warning message including information informing the manager or store clerk that there is a customer who has not registered the product 10 to the device or terminal of the manager or store clerk.

[0097] As described above, in the monitoring device 100 of this embodiment, before the transaction process is completed, the determination unit 108 determines whether the product 10 taken by the customer is included in the list of products read by the self-checkout terminal 50 in the subsequent product registration process, and if the processing unit 106 determines that the product 10 is not included in the list, it causes the output device to output information prompting further product registration processing.

[0098] According to this monitoring device 100, after detecting that a customer has taken a product 10 displayed near the self-checkout terminal 50, if the product 10 is not included in the list of registered products, a message urging the customer to register the product can be displayed on the touch panel 52 of the self-checkout terminal 50. In this way, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also effectively prevents the customer from missing out on checking out products 10 in the product display area 12.

[0099] Furthermore, in the monitoring device 100 of this embodiment, the processing unit 106 outputs an alert to an output device when it is determined that the product 10 is not included in the list. According to this monitoring device 100, after detecting that a customer has taken a product 10 displayed near the self-checkout terminal 50, if the product 10 is not included in the list of registered products, an alert can be output not only to the self-checkout terminal 50 used by the customer, but also to in-store signage, a monitoring device in the monitoring center, a manager's terminal, a store clerk's terminal, etc. In this way, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also notifies a manager or store clerk that an item has been omitted from the transaction, allowing the clerk to take appropriate action, such as directly speaking to the customer and encouraging them to register the product.

[0100] (Third Embodiment) This embodiment is similar to any of the above-described embodiments, except that, when it is detected that a customer has picked up a product during product registration processing, information prompting the customer to register the product is output. The monitoring device 100 of this embodiment will be described below using FIG. 1 as having the same configuration as the monitoring device 100 of FIG. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0101] <Example of functional configuration> The detection unit 104 processes the image to detect that the customer has picked up the product 10 during the product registration process. When the processing unit 106 detects that the customer has picked up the product 10, it causes the output device to output information prompting the customer to register the product 10 during the product registration process.

[0102] The processing of this processing unit 106 is the same as the processing (a) of the above embodiment. For example, in this embodiment, because the customer is performing a product registration operation at the self-checkout terminal 50, a product registration processing screen is displayed on the touch panel 52 of the self-checkout terminal 50. In this state, it is preferable to display information including at least one of the following: a message urging the customer to register the product 10 that the customer has picked up, a message informing the customer that the product 10 has not been registered, and a message informing the customer that the product 10 needs to be registered.

[0103] The criteria for determining whether or not a product registration process is in progress include the period during which the self-checkout terminal 50 is accepting product registration operations by a customer. In other words, the period during which a screen for accepting product registration is displayed on the operation screen of the self-checkout terminal 50. The period during which a product registration process is in progress is the period up until the time when the checkout button to proceed to the checkout process is accepted on that screen.

[0104] 11 is a flowchart showing an example of operation of the monitoring device 100 according to the embodiment. This flow includes the same steps as the flow in Fig. 2, and further includes step S301 between step S103 and step S105 in the flow in Fig. 2.

[0105] In this example, if the detection unit 104 detects in step S103 that the customer has picked up the product 10 (YES in step S103), the processing unit 106 determines whether the customer is currently registering the product at the self-checkout terminal 50 (step S301). If the customer is currently registering the product (YES in step S301), the process proceeds to step S109, where the processing unit 106 outputs information prompting the customer to register the product 10 to the output device (step S109). In other words, while a screen for registering the product is displayed on the touch panel 52 of the self-checkout terminal 50, the processing unit 106 displays information on the screen, including content prompting the customer to register the product 10 that the customer picked up.

[0106] On the other hand, if product registration processing is not in progress (NO in step S301), for example, if accumulating processing or payment processing is in progress, proceed to step S105. If the transaction processing has not yet been completed (NO in step S105), unlike in FIG. 2, return to step S301. In other words, wait until the transaction processing is completed. Then, if the transaction processing has been completed (YES in step S105), information that can identify the customer is stored in storage device 120 (step S107).

[0107] As described above, in the monitoring device 100 of this embodiment, the detection unit 104 processes the image to detect that a customer has picked up the product 10 during the product registration process, and when the processing unit 106 detects that the customer has picked up the product 10, it causes the output device to output information prompting the customer to register the product 10 during the product registration process.

[0108] According to this monitoring device 100, while a customer is registering a product, a message urging the customer to register the product 10 that the customer has taken from the product display area 12 can be displayed on the operation screen of the self-checkout terminal 50 performing the product registration process. In this way, the monitoring device 100 of this embodiment not only achieves the same effects as the above embodiment, but also effectively prevents the customer from forgetting to check out the product 10 in the product display area 12.

[0109] (Fourth Embodiment) FIG. 12 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to any of the above-described embodiments, except that it has a configuration in which, after detecting that a customer has picked up a product, it is determined by image processing whether or not product registration processing for that product has been performed. The monitoring device 100 according to this embodiment further includes an identification unit 110 in addition to the configuration of the monitoring device 100 of FIG. 1. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 of FIG. 1, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments as long as no contradictions arise.

[0110] <Example of Functional Configuration> Before the transaction process is completed, the identification unit 110 processes the image after the customer picks up the product 10 and determines that the product registration process has been performed for the product 10 when the relative distance between the position of the product registration reading device (scanner 54) and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion. The processing unit 106 outputs an alert to the output device if the product registration process has ended without the identification unit 110 determining that the product 10 has been registered. This process is the same as process (c) above.

[0111] Here, the method for determining whether or not a transaction has been completed before the transaction has been completed may be the same as the method described in the above embodiment.

[0112] The "end of product registration processing" occurs, for example, when the payment button on the product registration screen of the self-checkout terminal 50 is pressed.

[0113] The output device includes at least one of the touch panel 52 of the self-checkout terminal 50, a speaker of the self-checkout terminal 50, signage, a monitoring device in the monitoring center, a manager's terminal, and a store clerk's terminal.

[0114] Furthermore, after the transaction process is completed, the processing unit 106 may further cause an output device in the store to output information prompting the customer to perform the registration process for the product 10. This process is also referred to as process (d).

[0115] However, even before the transaction process is completed, the processing unit 106 may perform process (d) instead of or in addition to outputting the alert in process (c). The in-store output device may be, for example, at least one signage device installed in a store separate from the location of the self-checkout terminal 50. Alternatively, the in-store output device may include at least one speaker installed in the store.

[0116] The identification unit 110 causes the image processing device 130 to process the image after it is detected that the customer has taken the product 10. The image processing device 130 identifies the relative distance within the image between the position of the scanner 54 of the self-checkout terminal 50 and the position of at least one of the customer's fingers and the product 10. The criteria include that the relative distance is within a predetermined range. For example, the predetermined range is a distance within the image that corresponds to a few centimeters to 10 centimeters in the real world.

[0117] Preferably, the criterion is that the relative distances within the image between the position of the scanner 54 of the self-checkout terminal 50, the position of the customer's fingers, and the position of the product 10 are within a predetermined range. In other words, the criterion is that the scanner 54 of the self-checkout terminal 50, the customer's fingers, and the product 10 are all included within the predetermined range.

[0118] When registering a product 10, a customer brings the barcode of the product 10 close to the scanner 54's reading surface to have it read. At this time, the product 10 and the scanner 54 are spaced a certain distance apart. The reading distance DOF (Depth of Field) of the scanner 54 varies depending on the type of scanner 54 and the type of barcode (e.g., JAN, CODE39, etc.). The reading distance DOF also has a minimum and maximum value. Therefore, it is preferable that the relative distance be within a range between the minimum and maximum values. However, since the maximum value of the reading distance DOF can be several tens of centimeters, using the maximum value as the upper limit of the range makes it impossible to determine whether the barcode of the product 10 has been read. Therefore, it is preferable not to use the maximum value of the reading distance DOF as the upper limit of the predetermined range. The predetermined range may also use the average value of the distance at which the customer uses the scanner 54 to read the barcode of the product 10.

[0119] <Method of Setting Criteria> Furthermore, since the type of barcode varies depending on the product, it is preferable to set a criterion that matches the type of barcode attached to the product 10 displayed in the product display area 12. When multiple types of products 10 with different types of barcodes are displayed in the product display area 12, the product display area 12 is divided according to the type of barcode attached to the product 10. Then, the type of barcode is registered for each divided product display area 12. Furthermore, a criterion is associated with each barcode type and registered. The identification unit 110 may identify the type of barcode corresponding to the product display area 12 of the product 10 picked up by the customer, and obtain a criterion (relative distance range) corresponding to the identified barcode.

[0120] In the second embodiment, whether or not the product registration process for the product 10 has been completed is determined based on whether or not the POS data for the product 10 is included in the list of products registered at the self-checkout terminal 50. In contrast, in the present embodiment, the POS data does not need to be used. Also, it is not necessary to identify the product name or product code of the product 10 picked up by the customer.

[0121] After it is detected that a customer has taken a product 10 from the product display area 12, it is determined whether or not an action of bringing the product 10 close to the scanner 54 of the self-checkout terminal 50 is detected, that is, when the relative distance between the position of the scanner 54 and the position of the product 10 or the customer's fingers becomes less than a threshold, it is determined that the product registration process for the product 10 has been performed.

[0122] There is a possibility that the same product 10 displayed in the product display area 12 may be displayed on another display shelf in the store. In such a case, the product displayed on the display shelf and the product 10 displayed in the product display area 12 cannot be distinguished from each other based on the product code. In other words, even if the POS data for a product 10 is included in the product list, that product is not necessarily the product 10 that was displayed in the product display area 12. It is possible that a product taken from a display shelf is scanned, but a product taken from the product display area 12 is forgotten to be scanned. In such a case, it is not possible to detect an omission of a product taken from the product display area 12.

[0123] On the other hand, according to the configuration of this embodiment, it is possible to determine whether the product 10 actually picked up by the customer has been registered by bringing it close to the scanner 54, and therefore it is possible to distinguish between products taken from the display shelf and products taken from the product display area 12 and determine whether the product has been registered.

[0124] 13 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. This flow is started when the detection unit 104 detects that the customer has taken the product 10 in step S103 of FIG.

[0125] When the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), it sets a determination flag used to determine whether the customer's behavior is fraudulent to 1. The determination flag may be stored in the detection result 200 described above. Fig. 14 is a diagram showing an example of the data structure of the detection result 200 in which the determination flag is stored.

[0126] If the transaction has been completed (YES in step S105), the processing unit 106 proceeds to step S107 in FIG. 2 and executes process (b). That is, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107). On the other hand, if the transaction has not been completed (NO in step S105), the processing unit 106 executes process (a). That is, the processing unit 106 outputs information to the output device that prompts the user to register the product (step S109).

[0127] After step S109, if the transaction process has not yet been completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0128] If it is determined that the criteria are met (YES in step S403), the identification unit 110 determines that the product registration process for the product 10 has been performed, and resets the corresponding determination flag in the detection result 200 to 0 (step S409).Then, the process returns to step S101 in FIG.

[0129] On the other hand, if it is determined that the criteria are not met (NO in step S403), the processing unit 106 determines whether the product registration process is complete (step S405). If it is determined that the product registration process is complete (YES in step S405), the processing unit 106 executes process (c) (step S407). That is, the processing unit 106 causes the output device to output an alert.

[0130] However, the processing unit 106 may perform the above-described process (d) in step S407. The processing unit 106 can perform at least one of the processes (c) and (d).

[0131] On the other hand, if it is determined that the product registration process is not completed (NO in step S405), the process returns to step S105. In other words, during the product registration process, the processes in steps S105 to S403 are repeated to monitor whether the registration process of the product 10 is being performed.

[0132] Furthermore, processing unit 106 does not perform the process (b) of step S107, the process (a) of step S109, and the process (c) of step S407 when the determination flag is 0. In a configuration using a determination flag, the processes of steps S101 to S103, steps S105 to S109, steps S403 to S409, and steps S405 to S407 may be executed independently.

[0133] The processing unit 106 constantly performs the process of detecting whether the customer has taken the product 10 by the detection unit 104 in the processes of steps S101 to S103, and after setting the determination flag in step S401, executes each process. At this time, after the determinations of steps S105 and S405, the processing unit 106 refers to the determination flag to determine whether to execute the process (b) of step S107, the process (a) of step S109, and the process (c) of step S407. In other words, if the determination flag is set to 1, fraud is suspected and each process is executed. If the determination flag is set to 0, each process is not executed.

[0134] As described above, in the monitoring device 100 of this embodiment, before the transaction process is completed, the identification unit 110 processes the image after the customer picks up the product 10 and determines that the product registration process has been performed for that product 10 when the relative distance between the position of the product registration reading device (scanner 54) and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion.The processing unit 106 then causes the output device to output an alert if the identification unit 110 does not determine that the product registration process has been performed for the product 10 and the product registration process has ended.In addition, after the transaction process has been completed, the processing unit 106 also causes an output device in the store to output information prompting the customer to perform the registration process for that product 10.

[0135] With this monitoring device 100, it is not necessary to identify the product code or product name of the product 10 picked up by the customer, nor is it necessary to use POS data for registered products, in order to determine whether the product 10 picked up by the customer has been registered. The monitoring device 100 can determine that the product has been registered by bringing the product 10 actually picked up by the customer close to the scanner 54. Furthermore, if the product 10 has not been registered, an alert can be output when the product registration process has ended before the transaction is completed, to notify the customer that the product 10 has not been registered.

[0136] In this way, the monitoring device 100 of this embodiment achieves the same effects as the above embodiment, and furthermore, it can distinguish between products that customers have taken from display shelves and products that they have taken from the product display area 12, and determine whether or not the products have been registered, thereby efficiently preventing customers from missing out on accounting for products 10 that they have taken from the product display area 12.

[0137] Furthermore, when the product registration process has finished before the transaction process is completed, an alert can be output to the customer standing in front of the self-checkout terminal 50 to notify them that the product 10 has not been registered, and when the transaction process has been completed and the customer has left the self-checkout terminal 50, a message urging the customer to register the product 10 can be displayed on signage or the like within the store, so that the customer can be made aware of the missed transaction before leaving the store.

[0138] Fifth Embodiment Fig. 15 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to any of the above-described embodiments, except that it has a configuration in which the position of a scanner used in product registration processing or a product, etc., within an image is identified and registered through image processing. The monitoring device 100 according to this embodiment further includes a registration unit 112 in addition to the configuration of the monitoring device 100 shown in Fig. 12. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 shown in Fig. 12, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments as long as no contradictions arise.

[0139] <Example of Functional Configuration> The registration unit 112 processes the image to identify and pre-register the positions of at least one of the scanner 54, the product display area 12, and the products 10 within the image.

[0140] The detection unit 104 uses image processing to detect that the customer's hand 60 has entered the pre-registered product display area 12, and then detects that the product 10 has been taken by detecting that the product 10 is in the customer's hand 60. Furthermore, the identification unit 110 uses image processing to detect that the customer's hand 60 and the product 10 are at the pre-registered position of the scanner 54, and thereby identifies that the product 10 has been scanned.

[0141] <Operation Example> Fig. 16 is a flowchart showing an operation example of the monitoring device 100 according to the embodiment. Fig. 17 is a diagram for explaining a registration method performed by the registration unit 112. The following description will be given with reference to Figs. 16 and 17.

[0142] First, the image acquisition unit 102 acquires an image including the product display area 12 around the self-checkout terminal 50 (step S101). Then, the registration unit 112 displays the image 300 acquired in step S101 on the display device of the monitoring device 100, prompts the operator to mark the position of the product display area 12 of the product 10 to be monitored and the position of the scanner 54, and accepts the marking operation (step S601). Fig. 17(a) shows the state after the product display area 12 has been marked.

[0143] The registration unit 112 causes the image processing device 130 to perform image processing based on the positions of the received markings 310, thereby identifying the product display area position 302 and the scan position 304 (step S603). Alternatively, the registration unit 112 may individually identify the position of each product 10 through image processing by the image processing device 130. As shown in FIG. 17( b), the frames indicating the identified product display area position 302 and the scan position 304 are each displayed with a different line type. For example, the line type may be different colors, different thicknesses, or different line types (e.g., solid lines, dashed lines, dot-dash lines, etc.). Alternatively, the frame may be shaded with different types or colors of shading. The frames indicating the identified product display area position 302 and the scan position 304 are superimposed on the image 300.

[0144] The specified result is displayed on the screen, and confirmation is requested from the operator (step S605). If there is a change, the operator's operation to move or change the size of position 302 or position 304 is accepted, and the registration unit 112 changes position 302 or position 304 in accordance with the accepted content. Here, position 302 and position 304 are, for example, ranges indicated by coordinates.

[0145] Alternatively, the marking by the operator in step S601 may be omitted, and the positions of the product display area 12, the scanner 54, and the products 10 may be identified in step S603 through image recognition processing by the image processing device 130.

[0146] The registration unit 112 then registers the positions of at least one of the scanner 54, the product display area 12, and the product 10 in the image 300 as the position information 210 in the storage device 120 (step S607).

[0147] However, if the scanner 54 is a handheld scanner, even if its location is registered in advance, it may move, so it is preferable to identify the scanner's location through image recognition processing. Therefore, in the case of a handheld scanner, the configuration of the fourth embodiment described above determines whether the registration process for the product 10 has been completed based on whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion. Therefore, for handheld scanners, the prior location registration of this embodiment is not necessary.

[0148] However, if the location where the handheld scanner is placed is fixed, the location of the handheld scanner may be registered, since the process of determining the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 can be started after detecting whether the customer has placed their hand 60 on the handheld scanner.

[0149] In this embodiment, the location registration process is updated by a store clerk or the like when there is a change in the location of the product 10 or the scanner 54. Furthermore, by using an image captured by the camera 5 and having the image processing device 130 process the image, it may be possible to detect a change in the location of at least one of the product 10 and the scanner 54. When a change is detected, the registered location information may be updated. There are no particular limitations on the method for detecting a change in location using image processing, but it may be possible, for example, to detect the change by detecting a difference from a background image.

[0150] The position of the product display area 12 in the image may be predetermined. As described above, the position may be determined by marking by an operator, or the position may be determined in advance by image processing. The detection unit 104 processes the image of this position to detect that a customer has taken a product 10 from the product display area 12.

[0151] In this way, if the position of the product display area 12 within the image is predetermined, it is only necessary to process the image at that position, which reduces the load on image processing and also makes it possible to improve image recognition accuracy, for example, by performing image recognition processing appropriate for the image of the product display area 12.

[0152] As described above, in the monitoring device 100 of this embodiment, the registration unit 112 processes the image to identify and pre-register the positions of at least one of the scanner 54, the product display area 12, and the product 10 within the image.

[0153] In this way, the monitoring device 100 of this embodiment achieves the same effects as the above-described embodiment, and furthermore, since the positions of the product 10 to be detected and the scanner 54 are registered in advance, the area to be subjected to image recognition can be limited, thereby reducing the load on image processing and improving recognition accuracy.

[0154] Sixth Embodiment In this embodiment, a minimum configuration example will be described in which, when it is detected that a customer has picked up a product during product registration processing, information prompting the customer to register the product is output. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0155] <Example of Functional Configuration> The following will be described with reference to FIG. 1 . The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 is operated by a customer and acquires an image including a product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50) that has product registration and checkout functions. The detection unit 104 processes the image to detect that the customer has taken a product 10 from the product display area 12. When the detection unit 104 detects that the customer has taken a product 10, the processing unit 106 causes an output device to output information prompting the customer to register the product 10 during the product registration process.

[0156] The image acquisition unit 102 and the detection unit 104 are the same as those in the monitoring device 100 of the above-described embodiment.

[0157] 18 is a flowchart showing an example of operation of the monitoring device 100 of this embodiment. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). Then, the detection unit 104 determines whether product registration processing is in progress (step S301). Here, it can be determined that product registration processing is in progress while the operation screen for performing product registration processing is displayed on the self-checkout terminal 50.

[0158] If the product registration process is in progress (YES in step S301), the detection unit 104 causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that the customer has taken the product 10 from the product display area 12 (step S103). If the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), the processing unit 106 causes the output device to output information prompting the customer to register the product 10 during the product registration process (step S107).

[0159] On the other hand, if the product registration process is not in progress (NO in step S301), for example, before the product registration process, during the accumulation process, or during the payment process, the process returns to step S101. Also, if the detection unit 104 does not detect that the customer has taken the product 10 (NO in step S103), the process returns to step S101.

[0160] According to this monitoring device 100, the image acquisition unit 102 acquires an image including a product display area 12 within a predetermined range from the self-checkout terminal 50, the detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12, and when the detection unit 104 detects that a customer has taken a product 10, the processing unit 106 causes an output device to output information prompting the customer to register the product 10 during the product registration process.

[0161] Specifically, when a customer is in the process of registering a product at the self-checkout terminal 50 and the detection unit 104 detects that the customer has picked up a product 10 that is located near the self-checkout terminal 50, the processing unit 106 can display information on the touch panel 52 of the self-checkout terminal 50 that is in the process of registering the product, prompting the customer to register the picked-up product 10, or output the information as audio from the speaker of the self-checkout terminal 50.

[0162] In this way, the monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0163] Seventh Embodiment In this embodiment, a minimum configuration example will be described in which, after detecting that a customer has picked up a product, image processing is performed to determine whether product registration processing for the product has been performed. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0164] <Example of Functional Configuration> The following describes the functional configuration using FIG. 12 . The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, a processing unit 106, and an identification unit 110. The image acquisition unit 102 acquires an image including a product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50) operated by a customer and having product registration and checkout functions. The detection unit 104 processes the image to detect that the customer has taken a product 10 from the product display area 12. If the checkout process has not yet been completed, the identification unit 110 processes the image after the customer has taken the product 10 and determines that product registration processing has been performed for the product 10 when the relative distance between the position of the product registration reading device and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion. The processing unit 106 outputs an alert to the output device (process (c)) if the identification unit 110 does not determine that product registration processing has been performed for the product 10 and the product registration process has ended. 19 is a flowchart showing an example of operation of the monitoring device 100 of this embodiment. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). Then, the detection unit 104 causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that a customer has taken a product 10 from the product display area 12 (step S103).

[0165] If the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), the identification unit 110 determines whether the relative distance between the position of the reading means (scanner 54) for product registration and the position of at least one of the customer's fingers and the product 10 satisfies a criterion by having the image processing device 130 process the image after the customer has taken the product 10 (YES in step S105) (step S403).

[0166] If it is determined that the criteria are met (YES in step S403), the identification unit 110 determines that the product registration process for the product 10 has been completed, and returns to step S101. That is, the identification unit 110 continues to monitor whether the next product 10 is taken from the product display area 12 by the customer.

[0167] On the other hand, if it is determined that the criteria are not met (NO in step S403), the processing unit 106 determines whether the product registration process is complete (step S405). If it is determined that the product registration process is complete (YES in step S405), the processing unit 106 executes the above process (c) (step S407). That is, the processing unit 106 causes the output device to output an alert.

[0168] On the other hand, if it is determined that the product registration process is not completed (NO in step S405), the process returns to step S101. That is, the process continues to monitor whether the next product 10 is taken from the product display area 12 by the customer.

[0169] According to this monitoring device 100, the image acquisition unit 102 acquires an image including a product display area 12 within a predetermined range from the self-checkout terminal 50, the detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12, and the identification unit 110 processes the image after the customer has taken the product 10, if the transaction process has not yet been completed, and identifies that product registration processing has been performed for the product 10 when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product 10 meets a criterion, and the processing unit 106 outputs an alert to the output device on the condition that the product registration process has been completed without the identification unit 110 identifying that product registration processing has been performed for the product 10.

[0170] With this monitoring device 100, it is not necessary to identify the product code or product name of the product 10 picked up by the customer, nor is it necessary to use POS data for registered products, in order to determine whether the product 10 picked up by the customer has been registered. The monitoring device 100 can determine that the product has been registered by bringing the product 10 actually picked up by the customer close to the scanner 54. Furthermore, if the product 10 has not been registered, an alert can be output when the product registration process has ended before the transaction is completed, to notify the customer that the product 10 has not been registered.

[0171] In this way, the monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0172] While the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. For example, when the detection unit 104 detects that a customer has picked up a product 10, the processing unit 106 may output an output urging the customer to register the product 10, while the processing unit 106 may not display a checkout button for proceeding to the checkout process on the operation screen for performing the product registration process on the self-checkout terminal 50, or may display the checkout button as inactive so that pressing the checkout button is not accepted.

[0173] First, an example of the operation when a customer performs product registration processing and totaling processing at the self-checkout terminal 50 will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of the procedure for product registration processing and totaling processing at the self-checkout terminal 50.

[0174] First, a customer stands in front of the self-checkout terminal 50 (Step S1). Then, the customer places the product to be purchased or a shopping basket containing the product on the counter (Step S3). Then, the customer begins operating the self-checkout terminal 50 (Step S5). For example, at this time, a standby screen is displayed on the touch panel 52 of the self-checkout terminal 50. When the customer touches the touch panel 52, an initial screen for the product registration process is displayed.

[0175] The initial screen displays an operation screen for selecting whether or not a shopping bag is required, and the customer selects whether or not a shopping bag is required (Step S7). However, Step S7 may be omitted. After selecting the shopping bag, the screen transitions to the product registration processing screen, where the customer reads the product's barcode with the scanner 54 (Step S9). The product whose barcode was read is registered. The customer then places the scanned product into the shopping bag (or a shopping bag they brought with them) (Step S11).

[0176] Once all items have been scanned, the customer presses the payment button (step S13). The screen then transitions to a payment method selection screen, where the customer selects a payment method (step S15). The customer then pays using the selected payment method (step S17). The customer receives any change and a receipt (step S19). The customer then takes the items and leaves the store (step S21). The customer then exits the store (step S23).

[0177] When a customer performs the steps shown in Figure 20 in front of the self-checkout terminal 50, specific examples of the output content and output timing to prevent impulse purchases from being missed will be explained for each of the above embodiments.

[0178] Example 1 Fig. 21 is a diagram showing an example of operation of Example 1. In Example 1, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 2 of the first embodiment. After a customer stands in front of the self-checkout terminal 50 (step S1), performs a product registration process at the self-checkout terminal 50 (step S9), performs a payment process (step S13), makes a payment (step S17), and receives change and a receipt (step S19), if the detection unit 104 detects that the customer has taken a product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0179] Also, after the customer receives the change and receipt (step S19), the processing unit 106 stores information in the memory device 120 that can identify the customer who was detected by the detection unit 104 to have taken the product 10 (step S107).

[0180] In this way, in the first embodiment, it is possible to effectively prevent the omission of a transaction for a product 10 that is located near the self-checkout terminal 50. In addition, since it is possible to manage the information of customers who leave without registering the product, it is possible to take measures to prevent the omission of a transaction.

[0181] Example 2 Fig. 22 is a diagram illustrating an example of operation of Example 2. In Example 2, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 9 of Example 1 of the second embodiment. After a customer stands in front of the self-checkout terminal 50 (step S1), performs a product registration process at the self-checkout terminal 50 (step S9), performs a payment process (step S13), makes a payment (step S17), and receives the change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken a product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0182] On the other hand, if it is detected that the customer has taken product 10 from the product display area 12 before the transaction process is completed and it is determined that product 10 is not included in the list (NO in step S201), the processing unit 106 will further output information (message 410) to the output device prompting the product registration process (step S203).

[0183] Also, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the customer has taken product 10 and determines that product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0184] 23 to 25 are diagrams illustrating the timing of output of message 400 and message 410. In the example of FIG. 23, a list screen 450 of registered products in the product registration process is displayed on touch panel 52 of self-checkout terminal 50. The customer is then registering two products (step S9). During this product registration process, it is detected that the customer has removed product 10 (product C) from product display area 12 (step S103). Then, processing unit 106 causes message 400, including information prompting the customer to register product C, to be output on list screen 450 of registered products displayed on touch panel 52 of self-checkout terminal 50.

[0185] When the customer then presses the checkout button to proceed to the checkout process (step S13), the product registration process is completed, and the processing unit 106 then outputs a message 410 containing information urging the customer to register product C on the registered product list screen 450 displayed on the touch panel 52 of the self-checkout terminal 50 before proceeding to the checkout process. This message 410 is more forceful than the message 400 in urging the customer to scan product 10.

[0186] 24 shows examples of messages 402 and 412 that are output by processing unit 106 when detection unit 104 detects that a customer has taken a shopping bag before the transaction is completed. 25 shows examples of messages 404 and 414 that are output by processing unit 106 when detection unit 104 detects that a customer has taken a coffee cup for self-service coffee before the transaction is completed.

[0187] In this way, in Example 2, after it is detected that a customer has taken a product 10 from the product display area 12, it is checked whether the POS data of the product 10 is included in the list of registered products, and if the product 10 is not included in the list, information urging the customer to register the product 10 is output again, making it possible to more effectively prevent the failure to check out products 10 that are located near the self-checkout terminal 50. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent failure to check out products.

[0188] Example 3 Fig. 26 is a diagram showing an example of operation of Example 3. In Example 3, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 10 of Example 2 of the second embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken the product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0189] On the other hand, if it is detected that a customer has taken a product 10 from the product display area 12 before the transaction is completed and it is determined that the product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output an alert (step S221). For example, the processing unit 106 causes the clerk terminal 70 to output a message notifying the customer that there is a customer who has not registered the product 10 in the product display area 12. Alternatively, the processing unit 106 may cause the touch panel 52 of the self-checkout terminal 50 to display information including content strongly urging the customer to register the taken product 10, or may cause the speaker of the self-checkout terminal 50 to output an alert sound or message.

[0190] Also, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the customer has taken product 10 and determines that product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0191] In this way, in Example 3, after it is detected that a customer has taken a product 10 from the product display area 12, it is checked whether the POS data for the product 10 is included in the list of registered products, and if the product 10 is not included in the list, an alert is output to the output device, making it possible to more effectively prevent products 10 that are located near the self-checkout terminal 50 from being overlooked. In addition, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent overlooking of products.

[0192] Example 4 Fig. 27 is a diagram showing an example of operation of Example 4. In Example 4, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 13 of the fourth embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken the product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0193] Furthermore, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the customer has taken the product 10 and determines that the product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107). Furthermore, the processing unit 106 causes the output device to process an alert (message 430). For example, the processing unit 106 causes the output device to display the message 430 on the touch panel 52 of the self-checkout terminal 50 and to process an alarm sound and voice from the speaker of the self-checkout terminal 50. This message 430 is one level stronger than the message 400 and warns the customer to scan the product 10.

[0194] In this way, in Example 4, after it is detected that a customer has taken a product 10 from the product display area 12, information prompting the customer to register the product 10 is output, and if the customer then completes the transaction and leaves the self-checkout terminal 50, an alert can be output on in-store signage, etc., rather than from the touch panel 52 or speaker of the self-checkout terminal 50. In this way, the warning urging the customer to register the product can be continued even after the customer has left the self-checkout terminal 50, making it possible to more effectively prevent the customer from missing out on the transaction of products 10 that are located near the self-checkout terminal 50. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent customers from missing out on the transaction.

[0195] (Example 5) Fig. 28 is a diagram showing an example of operation of Example 5. In Example 5, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 11 of the third embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs processing to detect that the customer has taken a product 10 from the product display area 12 (step S103).

[0196] Then, between the time when the customer starts the product registration process (step S5) and the time when the customer presses the checkout button to proceed to the checkout process (step S13), that is, during the product registration process, if the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0197] On the other hand, after the customer receives the change and receipt (step S19), the processing unit 106 detects that the product 10 has been taken by the detection unit 104 and stores information that can identify the customer in the storage device 120 (step S107).

[0198] In this way, in Example 5, if it is detected that a customer has taken a product 10 from the product display area 12, during the product registration process, information urging the customer to register the product 10 can be displayed on the touch panel 52 of the self-checkout terminal 50 performing the product registration process, making it possible to more effectively prevent the customer from missing out on the transaction for products 10 that are located near the self-checkout terminal 50. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent the customer from missing out on the transaction.

[0199] Example 6 Fig. 29 is a diagram showing an example of operation of Example 6. In Example 6, a case will be described in which the monitoring device 100 operates in accordance with the flow of Fig. 13 of the fourth embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs processing to detect that the customer has taken a product 10 from the product display area 12 (step S103).

[0200] Then, when the detection unit 104 detects that the product 10 has been taken from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) urging the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109) until the product registration process is completed, that is, until the checkout button is pressed to move on to the checkout process (step S13).

[0201] Then, before the customer receives the change and receipt (step S19), i.e., before the transaction is completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0202] If it is determined that the criteria are not met (NO in step S403) and the product registration process is completed, that is, after the checkout button is pressed (step S13), the processing unit 106 outputs an alert to the output device (step S407). For example, the processing unit 106 displays a message 430 on the touch panel 52 of the self-checkout terminal 50 and controls the speaker of the self-checkout terminal 50 to output an alert sound and voice. This message 430 is one level stronger than the message 400, warning the customer to scan the product 10.

[0203] Also, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the product 10 has been taken and determines that the product 10 has not been scanned, the processing unit 106 stores information that can identify the customer in the memory device 120 (step S107).

[0204] In this way, in Example 5, after it is detected that a customer has taken a product 10 from the product display area 12, information prompting the customer to register the product 10 is output. If the customer does not scan the product 10 before the product registration process is completed, an alert can be output after the product registration process is completed. The alert can include alerts issued to the customer as well as to store clerks or managers. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent customers from missing out on transactions.

[0205] Example 7 Fig. 30 is a diagram showing an example of operation of Example 7. In Example 7, a case will be described in which the monitoring device 100 operates in accordance with the flow of Fig. 18 of the sixth embodiment. From after the customer starts the product registration process (step S5) until before the customer presses the checkout button to proceed to the checkout process (step S13), that is, during the product registration process, the detection unit 104 performs a process of detecting whether the customer has taken a product 10 from the product display area 12 (step S103).

[0206] During the product registration process, when the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0207] In this way, in Example 7, if it is detected that a customer has taken product 10 from the product display area 12 during the product registration process, information urging the customer to register product 10 can be displayed on the touch panel 52 of the self-checkout terminal 50 performing the product registration process, thereby making it possible to more effectively prevent products 10 stored near the self-checkout terminal 50 from being missed out on payment.

[0208] Example 8 Fig. 31 is a diagram showing an example of operation of Example 8. In Example 8, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 19 of the seventh embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs processing to detect that the customer has taken a product 10 from the product display area 12 (step S103).

[0209] When the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) prompting the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109).

[0210] Then, before the customer receives the change and receipt (step S19), i.e., before the transaction is completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0211] If it is determined that the criteria are not met (NO in step S403) and the product registration process is completed, that is, after the checkout button is pressed (step S13), the processing unit 106 outputs an alert to the output device (step S407). For example, the processing unit 106 displays a message 430 on the touch panel 52 of the self-checkout terminal 50 and controls the speaker of the self-checkout terminal 50 to output an alert sound and voice. This message 430 is one level stronger than the message 400, warning the customer to scan the product 10.

[0212] In this way, in the eighth embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, information prompting the customer to register the product 10 is output, and if the customer does not scan the product 10 before the product registration process is completed, an alert can be output after the product registration process is completed. The alert can include alerts to the customer, as well as alerts to store staff or managers.

[0213] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments can be combined as long as the content is not contradictory.

[0214] The present invention has been described above with reference to embodiments and examples, but the present invention is not limited to the above embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Note that, when information about users (customers) is acquired and used in the present invention, this must be done lawfully.

[0215] Some or all of the above embodiments can be described as, but are not limited to, the following notes: 1. A monitoring device comprising: image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device having product registration and checkout functions; detection means that detects that the customer has taken a product from the product display area by processing the image; and processing means that, when the detection means detects that the customer has taken the product, if the checkout process has not yet been completed, output information that prompts the customer to register the product to an output means; and if the checkout process has been completed, store information that can identify the customer in a storage means. 2. The monitoring device described in 1., wherein the information that can identify the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 3. 1. or 2. The monitoring device described in any one of items 1 to 3, further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product taken by the customer is included in a list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output information prompting the customer to proceed with further product registration process if it is determined that the product is not included in the list. 4. The monitoring device described in any one of items 1 to 3, further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product taken by the customer is included in a list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output an alert if it is determined that the product is not included in the list. 5. The monitoring device described in any one of items 1 to 4, further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product taken by the customer is included in a list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output an alert. In the monitoring device described in any one of the above, the detection means processes the image to detect that the customer has taken the product during the product registration process, and when the processing means detects that the customer has taken the product, the processing means causes the output means to output information prompting the customer to register the product during the product registration process.6. The monitoring device described in any one of 1. to 5., further comprising an identification means for identifying that product registration processing has been performed for a product before the transaction is completed, when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion by processing the image after the customer picks up the product, and the processing means causes the output means to output an alert if the identification means does not identify that the product registration processing has been performed for the product and the product registration processing has ended. 7. The monitoring device described in 6., wherein the processing means further causes an in-store output means to output information prompting the store to register the product after the transaction is completed. 8. The monitoring device described in 6. or 7., further comprising a registration means for processing the image to identify the position of the reading means in the image and register it in advance. 9. 6. to 8. 10. The monitoring device according to any one of items 6 to 9, further comprising a registration means for specifying and pre-registering the position of at least one of the product display area and the product in the image by processing the image. 11. The monitoring device according to any one of items 6 to 9, further comprising a registration means for specifying and pre-registering the position of the product display area and / or the product in the image by processing the image of this position.

[0216] 11. A monitoring system comprising: a monitoring device; a checkout device operated by a customer and having product registration and checkout functions; and a camera that generates an image including a product display area within a predetermined range of the checkout device, wherein the monitoring device has image acquisition means for acquiring the image; detection means for detecting that a customer has taken a product from the product display area by processing the image; and processing means for, when the detection means detects that the customer has taken the product, outputting information prompting the customer to register the product from an output means if the checkout process has not yet been completed, or storing information that can identify the customer in a storage means if the checkout process has been completed. 12. A monitoring system according to 11., wherein the information that can identify the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 13. 11. or 12. 14. The monitoring system described in any one of 11. to 13., wherein the monitoring device further has a determination means for determining, before the transaction is completed, whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process, and if it is determined that the product is not included in the list, the processing means of the monitoring device causes the output means to output information prompting the customer to perform further product registration processing. 15. The monitoring system described in any one of 11. to 13., wherein the monitoring device further has a determination means for determining, before the transaction is completed, whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process, and if it is determined that the product is not included in the list, the processing means of the monitoring device causes the output means to output an alert.15. A monitoring system described in any one of 11. to 14., wherein the detection means of the monitoring device processes the image to detect that the customer has picked up a product during product registration processing, and when the processing means of the monitoring device detects that the customer has picked up the product, it causes the output means to output information prompting the customer to register the product during the product registration processing. 16. A monitoring system described in any one of 11. to 15., wherein the monitoring device further comprises an identification means that, before the transaction processing is completed, processes the image after the customer picks up the product and, when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion, identifies that product registration processing has been performed for the product, and the processing means of the monitoring device causes the output means to output an alert on the condition that the product registration processing has ended without the identification means identifying that product registration processing has been performed for the product. 18. The monitoring system described in 16., wherein, after the transaction process is completed, the processing means of the monitoring device further outputs information to an output means in the store that prompts the customer to register the product. 19. The monitoring system described in any one of 16. to 18., wherein the monitoring device comprises registration means that processes the image to identify and pre-register the position of the reading means within the image. 20. The monitoring system described in any one of 16. to 19., wherein the position of the product display area within the image is predetermined, and the detection means of the monitoring device processes the image of this position to detect that the customer has taken the product from the product display area.

[0217] 21. A monitoring method in which one or more computers acquire an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions, process the image to detect that the customer has taken a product from the product display area, and when it is detected that the customer has taken the product, if the checkout process has not yet been completed, cause an output means to output information prompting the customer to register the product, or if the checkout process has been completed, cause a storage means to store information that can identify the customer. 22. A monitoring method in accordance with 21., wherein the information that can identify the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 23. 21. or 22. 24. The monitoring method described in any one of 21. to 23., wherein the one or more computers, before the transaction is completed, determine whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and if it is determined that the product is not included in the list, cause the output means to output information prompting the customer to continue the product registration process. 25. The monitoring method described in any one of 21. to 24., wherein the one or more computers, before the transaction is completed, determine whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and if it is determined that the product is not included in the list, cause the output means to output an alert. 26. The monitoring method described in any one of 21. to 24., wherein the one or more computers, by processing the image, detect that the customer has picked up a product during the product registration process, and if it is detected that the customer has picked up the product, cause the output means to output information prompting the customer to register the product during the product registration process.26. A monitoring method according to any one of 21. to 25., wherein, before the transaction is completed, the one or more computers process the image after the customer picks up the product, and when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product meets a criterion, determine that product registration processing has been performed for the product; and cause the output means to output an alert if the identification means does not determine that product registration processing has been performed for the product and the product registration processing has ended. 27. A monitoring method according to 26., wherein, after the transaction is completed, the one or more computers further cause an in-store output means to output information prompting the customer to register the product. 28. A monitoring method according to 26. or 27., wherein the one or more computers process the image to identify the position of the reading means in the image and register it in advance. 29. A monitoring method according to 26. to 28. 30. The monitoring method described in any one of 26. to 29., wherein the position of the product display area in the image is predetermined, and the one or more computers process the image of this position to detect that the customer has taken the product from the product display area.

[0218] 31. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions; detecting that the customer has taken a product from the product display area by processing the image; and, when it is detected in the detection step that the customer has taken the product, causing an output means to output information prompting the customer to register the product if the checkout process has not yet been completed; and storing information that can identify the customer in a storage means if the checkout process has been completed. 32. A computer-readable recording medium having recorded thereon a program in the recording medium described in 31., wherein the information that can identify the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 33. 31. or 32. 34. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: if the transaction has not yet been completed, in a subsequent product registration process, determining whether the product picked up by the customer is included in a list of products read by the settlement device; and if it is determined that the product is not included in the list, having the output means output information prompting the customer to continue the product registration process. 35. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: if the transaction has not yet been completed, in a subsequent product registration process, determining whether the product picked up by the customer is included in a list of products read by the settlement device; and if it is determined that the product is not included in the list, having the output means output an alert.35. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: 35. A recording medium described in any one of 31. to 34., comprising: a step of detecting that the customer has picked up a product during product registration processing by processing the image; and a step of causing the output means to output information prompting the customer to register the product during the product registration processing when it is detected that the customer has picked up the product. 36. A recording medium described in any one of 31. to 35., comprising: a step of determining that product registration processing has been performed for the product by processing the image after the customer has picked up the product when the relative distance between the position of a product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion; and a step of causing the output means to output an alert on the condition that the product registration processing has ended without the determination step determining that product registration processing for the product has been performed. 37. 36. 38. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedure: after the transaction process has been completed, outputting information to an output means in the store that prompts the user to register the product in question. 36. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedure: by processing the image, specifying the position of the reading means within the image and registering it in advance. 39. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedure: by processing the image, specifying the position of at least one of the product display area and the product within the image and registering it in advance.40. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps in the recording medium described in any one of 36. to 39.: the position of the product display area in the image is predetermined, and the image of this position is processed to detect that the customer has taken the product from the product display area.

[0219] 41. A program for causing a computer to execute the following steps: acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions; detecting that the customer has taken a product from the product display area by processing the image; and, when it is detected in the detection step that the customer has taken the product, outputting information from an output means prompting the customer to register the product if the checkout process has not yet been completed; and storing information that can identify the customer in a storage means if the checkout process has been completed. 42. A program according to 41., wherein the information that can identify the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 43. A program according to 41. or 42. 44. A program described in any one of paragraphs 41. to 43., a program causing a computer to execute the following steps: if the transaction has not yet been completed, in a subsequent product registration process, determining whether the product picked up by the customer is included in a list of products read by the settlement device; and if it is determined that the product is not included in the list, causing the processing means to output information to the output means prompting the customer to continue the product registration process. 44. A program described in any one of paragraphs 41. to 43., a program causing a computer to execute the following steps: if the transaction has not yet been completed, in a subsequent product registration process, determining whether the product picked up by the customer is included in a list of products read by the settlement device; and if it is determined that the product is not included in the list, causing the output means to output an alert. 45. A program described in any one of paragraphs 41. to 44., a program causing a computer to execute the following steps:46. ​​A program described in any one of 41. to 45., causing a computer to execute the following steps: if, before the transaction is completed, after the customer picks up the product, processing the image to determine that product registration processing has been performed for the product when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product meets a criterion; and if, by the determination step, the product registration processing is not determined to have been performed for the product and the product registration processing has ended, causing the output means to output an alert. 47. A program described in 46., causing a computer to execute the following steps: if the transaction is completed, further causing an output means in-store to output information prompting the customer to perform registration processing for the product. 48. A program described in 46. or 47., causing a computer to execute the following steps: processing the image to determine the position of the reading means within the image and register it in advance. 49. 46. to 48. 50. A program described in any one of items 46. to 49., which causes a computer to execute a procedure for specifying and pre-registering the position of at least one of the product display area and the product in the image by processing the image. 51. A program described in any one of items 46. to 49., which causes a computer to execute a procedure for detecting that the customer has taken the product from the product display area, wherein the position of the product display area in the image is predetermined, by processing the image of this position.

[0220] 51. A monitoring device comprising: image acquisition means for acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions; detection means for detecting that the customer has taken a product from the product display area during the product registration process by processing the image; and processing means for causing an output means to output information prompting the customer to register the product during the product registration process when it is detected that the customer has taken the product. 52. A monitoring device described in 51., wherein the information capable of identifying the customer includes at least one of identification information for the electronic payment medium used by the customer for the checkout process, an image including the customer, and human features extracted from the image. 53. 51. or 52. 54. The monitoring device described in any one of 51. to 53., further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output information prompting the customer to proceed with the product registration process if it is determined that the product is not included in the list. 55. The monitoring device described in any one of 51. to 54., further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output an alert if it is determined that the product is not included in the list. The monitoring device described in any one of the above further comprises an identification means for identifying that product registration processing has been performed for a product before the transaction process is completed, when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion by processing the image after the customer has picked up the product, and the processing means causes the output means to output an alert, on the condition that the product registration processing has ended without the identification means identifying that the product registration processing for the product has been performed.56. The monitoring device described in 55., wherein, after the transaction process is completed, the processing means further causes an in-store output means to output information prompting the customer to register the product. 57. The monitoring device described in 55. or 56., comprising registration means that processes the image to identify and pre-register the position of the reading means within the image. 58. The monitoring device described in any one of 55. to 57., comprising registration means that processes the image to identify and pre-register the position of at least one of the product display area and the product within the image. 59. The monitoring device described in any one of 55. to 58., wherein the position of the product display area within the image is predetermined, and the detection means processes the image of this position to detect that the customer has taken the product from the product display area.

[0221] 60. A monitoring system comprising: a monitoring device; a checkout device operated by a customer and having product registration and checkout functions; and a camera that generates an image including a product display area within a predetermined range from the checkout device, wherein the monitoring device comprises: image acquisition means that is operated by a customer and acquires an image including the product display area within a predetermined range from the checkout device operated by a customer and having product registration and checkout functions; detection means that processes the image to detect that a customer has taken a product from the product display area during the product registration process; and processing means that, when it is detected that the customer has taken the product, causes an output means to output information that prompts the customer to register the product during the product registration process. 61. A monitoring method in which one or more computers acquire an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions; process the image to detect that a customer has taken a product from the product display area during the product registration process; and, when it is detected that the customer has taken the product, causes an output means to output information that prompts the customer to register the product during the product registration process. 62. 63. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedures: acquiring an image including a product display area within a predetermined range from a checkout device that is operated by a customer and has product registration and checkout functions, processing the image to detect that the customer has taken a product from the product display area during the product registration process, and, when it is detected that the customer has taken the product, causing an output means to output information prompting the customer to register the product during the product registration process. 64. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedures: acquiring an image including a product display area within a predetermined range from a checkout device that is operated by a customer and has product registration and checkout functions, processing the image to detect that the customer has taken a product from the product display area during the product registration process, and, when it is detected that the customer has taken the product, causing an output means to output information prompting the customer to register the product during the product registration process.

[0222] 64. A monitoring device comprising: image acquisition means operated by a customer to acquire an image including a product display area within a predetermined range from a checkout device having product registration and checkout functions; detection means for processing the image to detect that the customer has taken a product from the product display area; identification means for processing the image after the customer has taken the product before the checkout process is completed and, when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion, identifying that product registration processing has been performed for the product; and processing means for causing output means to output an alert if the identification means does not identify that product registration processing has been performed for the product and the product registration process has ended. 65. The monitoring device described in 64., wherein the information capable of identifying the customer includes at least one of identification information for the electronic payment medium used by the customer in the checkout process, an image including the customer, and human features extracted from the image. 66. 64. or 65. 67. The monitoring device described in any one of 64. to 66., further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output information prompting the customer to further register the product if it is determined that the product is not included in the list. 68. The monitoring device described in any one of 64. to 67., further comprising a determination means for determining, if the transaction process has not yet been completed, whether the product picked up by the customer is included in the list of products read by the settlement device in the subsequent product registration process, and the processing means causes the output means to output an alert if it is determined that the product is not included in the list. In the monitoring device described in any one of the above, the detection means processes the image to detect that the customer has taken the product during the product registration process, and when the processing means detects that the customer has taken the product, the processing means causes the output means to output information prompting the customer to register the product during the product registration process.69. The monitoring device described in any one of 64. to 68., wherein, after the transaction process has been completed, the processing means further causes an in-store output means to output information prompting the customer to register the product. 70. The monitoring device described in any one of 64. to 69., comprising registration means that processes the image to identify and pre-register the position of the reading means within the image. 71. The monitoring device described in any one of 64. to 70., comprising registration means that processes the image to identify and pre-register the position of at least one of the product display area and the product within the image. 72. The monitoring device described in any one of 64. to 71., wherein the position of the product display area within the image is predetermined, and the detection means processes the image of this position to detect that the customer has taken the product from the product display area.

[0223] 73. A monitoring system comprising: a monitoring device; a checkout device operated by a customer and having product registration and payment functions; and a camera that generates an image including a product display area within a predetermined range from the checkout device, wherein the monitoring device comprises: image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from the checkout device and having product registration and payment functions; detection means that processes the image to detect that a customer has taken a product from the product display area; identification means that, after the customer has taken the product before the transaction is completed, processes the image to identify that product registration processing has been performed for the product when the relative distance between the position of a product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion; and processing means that causes an output means to output an alert if the identification means does not identify that product registration processing has been performed for the product and the product registration processing has ended. A monitoring method in which one or more computers acquire an image including a product display area within a predetermined range from a checkout device operated by a customer and having product registration and checkout functions, process the image to detect that the customer has taken a product from the product display area, process the image after the customer has taken the product before the checkout process is completed, and determine that product registration processing has been performed for the product when the relative distance between the position of a product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion, and output an alert to an output means on the condition that the product registration processing for the product has not been determined to have been performed and the product registration processing has ended.75. A program for causing a computer to execute the following steps: acquiring an image including a product display area within a predetermined range from a payment device that is operated by a customer and has product registration and payment functions; detecting that the customer has taken a product from the product display area by processing the image; specifying that product registration processing has been performed for the product by processing the image after the customer has taken the product before the transaction is completed, when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion; and outputting an alert to output means on the condition that the product registration processing for the product has not been determined to have been performed by the specifying step and the product registration processing has ended. 76. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: acquiring an image including a product display area within a predetermined range from a checkout device that is operated by a customer and has product registration and checkout functions; detecting that a customer has taken a product from the product display area by processing the image; determining that product registration processing has been performed for a product by processing the image after the customer has taken the product before the checkout process is completed, when the relative distance between the position of a product registration reading means and the position of at least one of the customer's fingers and the product satisfies a criterion; and outputting an alert to an output means, on the condition that the product registration processing for the product has not been determined to have been performed by the determination step and the product registration processing has ended.

[0224] REFERENCE SIGNS LIST 1 Surveillance system 3 Communication network 5 Camera 10 Product 12 Product display area 20 Shopping bag 30 Coffee cup 50 Self-checkout terminal 52 Touch panel 54 Scanner 60 Hand 70 Store clerk terminal 100 Monitoring device 102 Image acquisition unit 104 Detection unit 106 Processing unit 108 Determination unit 110 Identification unit 112 Registration unit 120 Storage device 130 Image processing device 200 Detection result 210 Location information 1000 Computer 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface

Claims

1. an image acquisition means for acquiring an image including a product display area within a predetermined range from a payment device having a product registration function and a payment function, the image acquisition means being operated by a customer; a detection means for detecting when a customer has removed an item from the product display area by processing the image; When the detection means detects that the customer has taken the product, If the transaction has not yet been completed, outputting information prompting the user to register the product to the output means; A monitoring device comprising: a processing means for storing information capable of identifying the customer in a storage means after the transaction process has been completed.

2. 2. The monitoring device according to claim 1, A monitoring device, wherein the information capable of identifying the customer includes at least one of identification information of the electronic payment medium used by the customer in the transaction, an image including the customer, and human features extracted from the image.

3. 3. The monitoring device according to claim 1, If the transaction has not yet been completed, the system further includes a determination unit for determining whether or not the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process, The processing means, when it is determined that the product is not included in the list, causes the output means to output information prompting further product registration processing.

4. The monitoring device according to any one of claims 1 to 3, If the transaction has not yet been completed, the system further includes a determination unit for determining whether or not the product selected by the customer is included in a list of products read by the settlement device in a subsequent product registration process, The monitoring device, wherein the processing means causes the output means to output an alert when it is determined that the product is not included in the list.

5. The monitoring device according to any one of claims 1 to 4, and, if the transaction has not yet been completed, further comprising: a means for processing the image after the customer has picked up the product, and when a relative distance between a position of a reading means for product registration and a position of at least one of the customer's fingers and the product satisfies a criterion, determining that product registration processing has been completed for the product; The processing means includes: The monitoring device causes the output means to output an alert on condition that the product registration process for the product is completed without the identification means identifying that the product registration process for the product has been performed.

6. 6. The monitoring device according to claim 5, a registration means for specifying and registering in advance a position of the reading means within the image by processing the image;

7. 7. The monitoring device according to claim 5, The monitoring device further comprises a registration means for specifying and registering in advance the position of at least one of the product display area and the product in the image by processing the image.

8. A monitoring device; a payment device operated by a customer and having a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the payment device; The monitoring device includes: image acquisition means for acquiring the image; a detection means for detecting when a customer has removed an item from the product display area by processing the image; When the detection means detects that the customer has taken the product, If the transaction has not yet been completed, outputting information prompting the user to register the product to the output means; A monitoring system having a processing means for storing information capable of identifying the customer in a storage means after the transaction process is completed.

9. One or more computers acquiring an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration function and a payment function; processing the images to detect when a customer has removed an item from the merchandise display area; When it is detected that the customer has taken the product, If the transaction has not yet been completed, outputting information prompting the user to register the product to the output means; A monitoring method in which, after the transaction process has been completed, information capable of identifying the customer is stored in a storage means.

10. On the computer, acquiring an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration function and a payment function; detecting when a customer has removed an item from the merchandise display area by processing the image; When it is detected in the detecting step that the customer has taken the product, a step of outputting information prompting the user to perform registration processing for the product in the case where the transaction processing has not yet been completed, to an output means; A program for executing a procedure for storing information capable of identifying the customer in a storage means after the transaction process is completed.