Monitoring system, monitoring device, monitoring method, and program

Abstract

To provide a monitoring system, a monitoring device, a monitoring method, and a program for solving the problem of missing an item in the accounting process in the vicinity of a checkout device that is operated by a customer and has the function of registering and checking out an item. [Solution] The monitoring device comprises an image acquisition unit that is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device that has product registration and checkout functions; a detection unit that detects that a customer has taken a product from the product display area by processing the image; and a processing unit that, when the detection unit detects that a customer has taken a product, outputs information to an output device prompting the customer to register the product if the checkout process has not yet been completed, or stores information that can identify the customer in a storage device if the checkout process has been completed.

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring device, a monitoring method, and a recording medium. [Background technology]

[0002] In recent years, retail stores and other businesses have been forced to reduce the number of employees in order to alleviate labor shortages and reduce labor costs. Therefore, so-called self-checkout registers, which allow customers to register products and process payments themselves, have become widespread. However, because self-checkout registers are operated by customers themselves, there is a risk of fraud by customers, and therefore countermeasures against fraud are necessary.

[0003] An example of a payment system that takes measures against such fraudulent activities is described in Patent Document 1. This payment system includes a clerk registration means through which a clerk registers items; a payment means operated by a customer to settle the payment; an imaging means for capturing images of the customer's operations as the payment is being made by the payment means; a display means for displaying the captured images; a state control means for controlling the clerk registration means between a first state in which the clerk mainly registers items and a second state other than the first state; and a display control means for changing the display control of the display means depending on the state controlled by the state control means. With this configuration, the payment system described in Patent Document 1 can capture a series of registration actions, such as the customer's actions of removing items from a shopping cart, registering items, and adding registered items to the shopping cart, on a wide-area imaging screen. This makes customers aware of the monitoring in full self-service mode, thereby deterring fraudulent activities related to payment.

[0004] Furthermore, Patent Documents 2 to 4 describe systems that automatically register products by identifying products taken from shelves and adding them to the customer's purchase list. In all of these documents, images captured by a camera and data detected by a weight sensor are used to detect that a product has been taken from a shelf based on the customer's position and movement, and the customer's movements within the store are tracked, thereby identifying and linking the product taken from the shelf with the customer who took it, and enabling automatic payment. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-157420 [Patent Document 2] Patent Publication No. 2021-108208 [Patent Document 3] Japanese Patent Publication No. 2020-115261 [Patent Document 4] International Publication No. 2021 / 125357 Summary of the Invention [Problem to be solved by the invention]

[0006] The technologies described in Patent Documents 2 to 4 relate to technology for automatically registering products and performing settlement. Therefore, the technologies described in Patent Documents 2 to 4 do not require customers to register products or perform settlement processes, so there is no risk of missing items or customers committing fraud during product registration or settlement processes. Furthermore, the technology described in Patent Document 1 is intended to prevent fraudulent acts during settlement, such as falsely or mistakenly reporting an amount that is higher than the amount inserted. Therefore, the technology described in Patent Document 1 cannot prevent missing items from being registered.

[0007] In response to this, the inventor focused on the problem that after a customer picks up a product near a payment device that is operated by the customer and has product registration and payment functions, the product may frequently be omitted from the payment, either intentionally or inadvertently.

[0008] In view of the above-mentioned problems, one example of the object of the present invention is to provide a monitoring system, a monitoring device, a monitoring method, and a recording medium that solve the problem of missing a certain product in the vicinity of a checkout device that is operated by a customer and has product registration and checkout functions. [Means for solving the problem]

[0009] According to one aspect of the present invention, an image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device having a product registration function and a checkout function; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detecting means detects that the customer has picked up the product, the processing unit detects that the customer has completed the transaction; outputting information prompting the user to register the product until completion of the transaction is detected; The monitoring device includes a processing means for storing information that can identify the customer in a storage means after the completion of the transaction process is detected.

[0010] According to one aspect of the present invention, A monitoring device; a payment device that is operated by a customer and has a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the checkout device; The monitoring device image acquisition means for acquiring the image; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detecting means detects that the customer has picked up the product, the processing unit detects that the customer has completed the transaction; outputting information prompting the user to register the product until completion of the transaction is detected; A monitoring system is provided that has a processing means that, after the completion of the transaction is detected, stores information that can identify the customer in a storage means.

[0011] According to one aspect of the present invention, One or more computers acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; When it is detected that the customer has picked up the product, a process is performed to detect the completion of the customer's checkout process; outputting information prompting the user to register the product until completion of the transaction is detected; A monitoring method is provided in which, after the completion of the transaction is detected, information that can identify the customer is stored in a storage means.

[0012] According to one aspect of the present invention, On the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; In the detecting step, when it is detected that the customer has picked up the product, a process is performed to detect completion of the customer's checkout process; outputting information prompting the user to register the product until completion of the transaction is detected; A program is provided for executing a procedure for storing information that can identify the customer in a storage means after the completion of the transaction process is detected.

[0013] Another aspect of the present invention may be a program that causes at least one computer to execute the method of the above aspect, or a computer-readable recording medium on which such a program is recorded. This recording medium includes a non-transitory tangible medium. The computer program comprises computer program code which, when executed by a computer, causes the computer to perform the monitoring method on a monitoring device.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention.

[0015] Furthermore, the various components of the present invention do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be part of another component, or part of one component may overlap with part of another component, etc.

[0016] Furthermore, although the method and computer program of the present invention describe a number of steps in a sequential order, the order in which the steps are described does not limit the order in which the steps are executed. Therefore, when implementing the method and computer program of the present invention, the order of the steps can be changed as long as it does not cause any problems in terms of the content.

[0017] Furthermore, the multiple steps of the method and computer program of the present invention are not limited to being executed at different times, and therefore, a step may occur while another step is being executed, or the execution timing of a step may partially or completely overlap with the execution timing of another step, etc. [Effects of the Invention]

[0018] According to one aspect of the present invention, a monitoring system, a monitoring device, a monitoring method, and a recording medium are provided that solve the problem of missing an item in the vicinity of a checkout device that is operated by a customer and has product registration and checkout functions. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an overview of a monitoring device according to an embodiment. [Figure 2] 10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of an image acquired by the monitoring device. [Figure 4] 1 is a diagram conceptually illustrating a system configuration of a monitoring system according to an embodiment. [Figure 5] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer that realizes the monitoring device. [Figure 6] FIG. 4 is a diagram illustrating an example of a data structure of a detection result according to an embodiment. [Figure 7] FIG. 2 is a functional block diagram illustrating an example of a logical configuration of a monitoring device according to an embodiment. [Figure 8] FIG. 4 is a diagram illustrating an example of a data structure of a detection result according to an embodiment. [Figure 9] 10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 10] 10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 11] 10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 12] FIG. 2 is a functional block diagram illustrating an example of a logical configuration of a monitoring device according to an embodiment. [Figure 13] 10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a data structure of a detection result in which a determination flag is stored. [Figure 15] FIG. 2 is a functional block diagram illustrating an example of a logical configuration of a monitoring device according to an embodiment. [Figure 16]10 is a flowchart illustrating an example of the operation of the monitoring device according to the embodiment. [Figure 17] FIG. 10 is a diagram for explaining a registration method performed by a registration unit. [Figure 18] 4 is a flowchart illustrating an example of the operation of the monitoring device of the present embodiment. [Figure 19] 4 is a flowchart illustrating an example of the operation of the monitoring device of the present embodiment. [Figure 20] 10A and 10B are diagrams illustrating an example of a procedure for product registration processing and payment processing at a self-checkout terminal. [Figure 21] FIG. 1 is a diagram illustrating an example of operation of the first embodiment. [Figure 22] FIG. 10 is a diagram illustrating an example of operation of the second embodiment. [Figure 23] FIG. 10 is a diagram illustrating the timing of message output. [Figure 24] FIG. 10 is a diagram illustrating the timing of message output. [Figure 25] FIG. 10 is a diagram illustrating the timing of message output. [Figure 26] FIG. 10 is a diagram illustrating an example of operation of the third embodiment. [Figure 27] FIG. 10 is a diagram illustrating an example of operation of the fourth embodiment. [Figure 28] FIG. 13 is a diagram illustrating an example of operation of the fifth embodiment. [Figure 29] FIG. 13 is a diagram illustrating an example of operation of the sixth embodiment. [Figure 30] FIG. 13 is a diagram illustrating an example of operation of the seventh embodiment. [Figure 31] FIG. 13 is a diagram illustrating an example of operation of the eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by similar reference numerals, and their description will be omitted where appropriate. In addition, in the following drawings, configurations of parts that are not related to the essence of the present invention are omitted and are not shown.

[0021] In the embodiments, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and inputting data or information output from another device into the device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. An example of passive acquisition is receiving information that is distributed (or transmitted, pushed, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0022] <Minimum configuration example> 1 is a diagram showing an overview of a monitoring device 100 according to an embodiment. The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having a product registration function and a payment settlement function. The detection unit 104 processes the image to detect that a customer has taken a product from the product display area. When the detection unit 104 detects that the customer has taken a product, the processing unit 106 performs the following process (a) and process (b). In process (a), if the transaction process has not yet been completed, information prompting the customer to register the product is output to the output device. In process (b), after the transaction has been completed, information that can identify the customer is stored in a storage device.

[0023] <Example of operation> Fig. 2 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. Fig. 3 is a diagram showing an example of an image captured of a customer making an impulse purchase while a product is being registered at a checkout device operated by the customer. The following description will be given with reference to Figs. 2 and 3.

[0024] First, the image acquisition unit 102 acquires an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration function and a payment settlement function (step S101). The detection unit 104 processes the image to detect that the customer has taken a product from the product display area (step S103).

[0025] As shown in Fig. 3(a), around the checkout device (self-checkout terminal 50) operated by the customer, promotional products 10 are displayed in a product display area 12 to encourage customers who are undergoing product registration processing to make an impulse purchase. Then, by using image processing, the detection unit 104 detects that the customer's hand 60 has entered the product display area in the image as shown in Fig. 3(b), and thereby detects that the customer may have taken a product or that the customer's hand 60 is holding a product.

[0026] When the detection unit 104 detects that the customer has taken a product (YES in step S103), the processing unit 106 performs the following process (a) and process (b). In process (a), if the transaction process has not yet been completed (NO in step S105), information prompting the user to register the product is output to the output device (step S109). In process (b), if the transaction process has been completed (YES in step S105), information that can identify the customer is stored in a storage device (step S107).

[0027] In the monitoring device 100, the image acquisition unit 102 acquires an image including a product display area within a predetermined range from a checkout device operated by a customer, and the detection unit 104 processes the image to detect that the customer has taken a product from the product display area. When the detection unit 104 detects that the customer has taken a product, the processing unit 106 performs the following processes (a) and (b). Process (a), if the transaction has not yet been completed, causes the output device to output information prompting the customer to register the product. Process (b), if the transaction has already been completed, causes the storage device to store information that can identify the customer.

[0028] According to this monitoring device 100, it is possible to obtain a monitoring system, a monitoring device, a monitoring method, and a recording medium that are operated by a customer and solve the problem of missing a certain product in the vicinity of a checkout device having product registration and checkout functions.

[0029] A detailed example of the monitoring device 100 will be described below.

[0030] (First embodiment) <System Overview> FIG. 4 is a diagram conceptually showing the system configuration of a monitoring system 1 according to an embodiment. The monitoring system 1 includes a monitoring device 100, at least one camera 5, and at least one self-checkout terminal 50, which are connected to the monitoring device 100 via a communication network 3. The monitoring device 100 includes a storage device 120. The storage device 120 may be provided inside the monitoring device 100 or may be provided externally. In other words, the storage device 120 may be hardware that is integrated with the monitoring device 100, or may be hardware that is separate from the monitoring device 100.

[0031] The self-checkout terminal 50 is a payment device operated by a customer and has product registration and payment functions. The self-checkout terminal 50 has at least a touch panel 52 and a scanner 54. The self-checkout terminal 50 may also include a printer that prints out receipts and the like. The touch panel 52 includes a display unit such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and the display unit and operation unit are integrated together.

[0032] Scanner 54 reads barcodes containing product information to register products purchased by customers. Scanner 54 may be a code reader equipped with a light source and a light-receiving element such as a photodiode to detect product codes for identifying products, or a camera equipped with a lens and an imaging element such as a CCD (Charge Coupled Device) image sensor to capture images including the product's exterior. Scanner 54 may be either a handheld type that a customer holds in their hand or a stationary type that reads product codes by bringing the scanner close to the reading surface.

[0033] The self-checkout terminal 50 may further include a magnetic card reader or an IC card reader that reads store membership cards, credit cards, and cash cards (magnetic cards or IC cards) from financial institutions, etc. Alternatively, the self-checkout terminal 50 may include an IC card reader that reads transportation IC cards and IC cards from mobile devices such as smartphones via contactless IC communication (NFC (Near Field Communication) communication). Furthermore, the self-checkout terminal 50 may include a reader that reads QR codes for QR (Quick Response) Code (registered trademark) payments. At least one of these readers and the scanner 54 may be integrated.

[0034] The camera 5 includes an imaging element such as a lens and a CCD (Charge Coupled Device) image sensor, and is, for example, a network camera such as an IP (Internet Protocol) camera. The network camera has, for example, a wireless LAN (Local Area Network) communication function and is connected to the monitoring device 100 via a communication network 3, i.e., a relay device (not shown) such as a router. These cameras 5 may be so-called monitoring cameras, with multiple cameras installed in a store or facility. The camera 5 may also include a mechanism for controlling the orientation of the camera body and lens, zooming, focusing, etc., in accordance with the movement of a person.

[0035] The images generated by the camera 5 are preferably taken in real time and transmitted to the monitoring device 100. However, the images transmitted to the monitoring device 100 do not have to be transmitted directly from the camera 5, but may be images delayed by a predetermined time. The images captured by the camera 5 may be temporarily stored in another storage device, and the monitoring device 100 may read them out from the storage device sequentially or at predetermined intervals. Furthermore, the images transmitted to the monitoring device 100 are preferably moving images, but may also be frame images taken at predetermined intervals or still images.

[0036] The monitoring device 100 is further connected to an image processing device 130. However, the image processing device 130 may be provided inside or outside the monitoring device 100. In other words, the image processing device 130 may be hardware that is integrated with the monitoring device 100, or may be hardware that is separate from the monitoring device 100.

[0037] The image processing device 130 performs image recognition processing on the image captured by the camera 5 to identify the positions of the product display area 12, the product 10, and the customer's hand 60 in the image. Furthermore, the image processing device 130 may track the movement of at least one of the product 10 and the customer's hand 60. Furthermore, the image processing device 130 may recognize that the customer's hand 60 has picked up or held the product 10. Furthermore, the image processing device 130 may register features of each product 10 (including plastic bags, cups for self-service drinks, and the following various items in the case where a fee is charged, such as chopsticks, forks, spoons, and straws) in advance, and identify the product 10 in the image by processing to compare the features of the product 10.

[0038] <Hardware configuration example> 5 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the monitoring device 100. The self-checkout terminal 50 and image processing device 130 in FIG. 4 are also realized by the computer 1000. Furthermore, the camera 5 in FIG. 4 may also be realized by the computer 1000. Furthermore, the functions of the monitoring device 100 may be shared and realized by at least two of the self-checkout terminal 50, the monitoring device 100, and the camera 5.

[0039] The computer 1000 includes a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0040] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0041] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0042] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0043] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the monitoring device 100 (e.g., the image acquisition unit 102, the detection unit 104, the processing unit 106, the determination unit 108 described below, the identification unit 110, the registration unit 112, etc.). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also store each piece of data from the storage device 120 of the monitoring device 100.

[0044] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded may include a non-transitory, tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0045] The input / output interface 1050 is an interface for connecting the computer 1000 with various input / output devices. The input / output interface 1050 also functions as a communication interface for performing short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication).

[0046] The network interface 1060 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1060 to the communication network may be a wireless connection or a wired connection.

[0047] The computer 1000 is connected to necessary devices (for example, the touch panel 52 of the self-checkout terminal 50, the scanner 54, a speaker, a printer, various readers, etc.) via the input / output interface 1050 or the network interface 1060.

[0048] The components of the monitoring device 100 in each of the embodiments shown in Figure 1 and Figures 7, 12, and 15 described below are realized by any combination of hardware and software of the computer 1000 in Figure 5. Those skilled in the art will understand that there are many variations in the realization methods and devices. The functional block diagrams showing the monitoring device 100 in each embodiment show logical functional blocks rather than a hardware-based configuration.

[0049] <Example of functional configuration> An example of the functional configuration of a monitoring device 100 according to an embodiment will be described with reference to FIG. The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit . The image acquisition unit 102 is operated by a customer and acquires an image including the product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50 in FIG. 4) having a product registration function and a checkout function.

[0050] The predetermined range is, for example, 2 meters or less. Alternatively, the predetermined range may be the range within reach of the hand 60 of a customer operating the self-checkout terminal 50 or the range within the customer's line of sight. In other words, the predetermined range is preferably a distance that allows a customer operating the self-checkout terminal 50 to easily notice the presence of the product 10 and pick it up. The product 10 is, for example, a product that is the subject of a sales promotion, and is displayed in the product display area 12 so as to encourage the customer operating the self-checkout terminal 50 to make an impulse purchase.

[0051] The products 10 also include plastic bags 20, which are subject to a fee, and coffee cups 30 for use in self-service coffee machines. The products 10 are provided with barcodes for product registration. However, the plastic bags 20 and coffee cups 30 do not necessarily have to have barcodes for product registration; for example, dedicated barcodes (or two-dimensional codes) may be provided near the plastic bags 20 and coffee cups 30. Alternatively, the use of plastic bags 20 and the type of coffee to be used with the coffee cups 30 may be registered on the product registration screen of the self-service checkout terminal 50.

[0052] The detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12. The detection unit 104 causes the image processing device 130 to process the image captured by the camera 5. The image processing device 130 uses image recognition processing to detect that a customer has taken a product 10 from the product display area 12. The detection method will be described in detail in the embodiments described below.

[0053] When the detection unit 104 detects that the customer has taken the product 10, the processing unit 106 performs the following process (a) and process (b). In process (a), if the transaction process has not yet been completed, information prompting the user to perform the registration process for the product 10 is output to the output device. In process (b), after the transaction process is completed, information that can identify the customer is stored in the storage device 120.

[0054] "Before the transaction is completed" refers to the period from the moment the customer stands in front of the self-checkout terminal 50 and the touch panel 52 accepts the pressing of the start transaction button on the standby screen displayed on the touch panel 52, including the period during product registration processing and payment procedures, until the customer has received their change and receipt. "Before the transaction is completed" may also refer to the period from the moment the customer stands in front of the self-checkout terminal 50 until the moment the customer leaves the self-checkout terminal 50.

[0055] The following are examples of methods for determining whether accounting has not yet been completed. At least one of the following determination conditions may be used, or at least two of them may be used in combination. (a1) While the image from the camera 5 is processed to detect that a customer is standing in front of the self-checkout terminal 50 (the customer detection process may be performed by, for example, the detection unit 104) (however, the presence of the customer may be detected by various human sensors such as infrared, visible light, piezoelectric, electrostatic, and ultrasonic sensors, but the sensor type is not limited to these). (a2) After the customer presses the checkout process start button on the standby screen displayed on the touch panel 52 of the self-checkout terminal 50 (a3) While the self-checkout terminal 50 accepts product registration operations by the customer (a4) While the self-checkout terminal 50 accepts the customer's payment processing operation (a5) While the self-checkout terminal 50 detects that the receipt and change have not been received by the customer

[0056] "After the transaction process is completed" refers to the period from when the customer receives their change and receipt at the self-checkout terminal 50 until they leave the store. "After the transaction process is completed" may also refer to the period from when the customer leaves the self-checkout terminal 50 until they leave the store.

[0057] The method for determining whether or not the transaction has been completed is exemplified below. At least one of the following determination conditions may be used, or at least two of them may be used in combination. (b1) After processing the image from the camera 5 and detecting that the customer in front of the self-checkout terminal 50 has left (b2) After the self-checkout terminal 50 detects that the receipt and change have been received by the customer (b3) While processing the image from camera 5 and tracking the customer in the store or detecting the presence of the customer in the store (b4) After processing the image from the camera 5 and detecting that the customer has left the self-checkout terminal 50, until detecting that the customer has left the store exit

[0058] The output device may include, for example, at least one of the touch panel 52 of the self-checkout terminal 50, a speaker of the self-checkout terminal 50, and signage or a speaker located near the self-checkout terminal 50, and is not particularly limited as long as the customer can confirm the output information on the spot. If the output device is a display device, it displays a screen displaying information prompting the customer to register the product. If the output device is a speaker, it outputs a message by voice prompting the customer to register the product. Alternatively, information notifying the customer that there is an unregistered product may be printed on the receipt output from the self-checkout terminal 50. In other words, the output device may be a printer of the self-checkout terminal 50. Furthermore, an electronic receipt including information notifying the customer that there is an unregistered product may be sent to a device such as a customer's smartphone on which an application capable of receiving electronic receipts (hereinafter referred to as an electronic receipt) has been pre-installed. In other words, the output device may include a customer's device such as a smartphone or tablet on which an application capable of receiving and displaying an electronic receipt has been installed.

[0059] For example, when a customer picks up a product with the product name "CCC Gum," the information to be output may be "Please scan the product (which may be the product name "CCC Gum")," "Did you forget to scan the product (which may be the product name "CCC Gum")?", "There is a product that has not been scanned," "CCC Gum is a product that needs to be scanned," etc. In other words, the information to be output includes at least one of the following: content urging the customer to register the product, content informing the customer that registration processing has not been performed for the product, and content informing the customer that registration processing for the product is required.

[0060] Furthermore, the information to be output may include content emphasizing that the customer has just picked up an item, such as "Please scan the item you just picked up," or "The item you just picked up needs to be scanned." This will make the customer aware that their actions are being monitored, which will likely deter fraudulent behavior.

[0061] Furthermore, as described above, when a customer takes an item including a shopping bag, a cup for self-service drinks (for example, a coffee cup), and the following various items that are subject to a fee, such as chopsticks, forks, spoons, and straws, information can be output in which the product name is replaced with the product name, such as a shopping bag or coffee cup (or self-service coffee). In the case of these items, the output information may be an operation screen that accepts the selection of use on the touch panel 52 of the self-checkout terminal 50.

[0062] For example, if it is detected that a customer has taken a shopping bag or coffee cup, the processing unit 106 may cause the touch panel 52 of the self-checkout terminal 50 to display an operation screen including information such as "Please enter the number of shopping bags you have taken," "There is a charge for the shopping bags you have taken. Please select the number you would like to use," and "Please select the type of coffee you would like to purchase," as well as a UI (User Interface) that accepts customer operations.

[0063] The information that can identify a customer includes at least one of the following: identification information of the electronic payment medium used by the customer for the transaction, a credit card number, a store membership card number (customer identification information), an image containing the customer, and human features extracted from the image.

[0064] The information that can identify the customer can be, for example, information acquired by the self-checkout terminal 50. Alternatively, the information that can identify the customer can be acquired by using an image captured by the camera 5 or by extracting features by processing the image processing device 130.

[0065] The image including the customer and the image from which the feature amount is extracted may be taken by a camera other than the camera 5, or may be taken by the camera 5 itself.

[0066] <Example of operation> An example of the operation of the monitoring device 100 will be described with reference to FIG. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). For example, the image acquired by the image acquisition unit 102 is exemplified in FIG. 3(a).

[0067] Then, the detection unit 104 causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that the customer has taken the product 10 from the product display area 12 (step S103). The image acquisition process of step S101 and the detection process of step S103 may be executed repeatedly while the customer is in front of the self-checkout terminal 50.

[0068] As shown in Fig. 3(a), around the checkout device (self-checkout terminal 50) operated by the customer, promotional products 10 are displayed in a product display area 12 to encourage customers who are undergoing product registration processing to make an impulse purchase. Then, as shown in Fig. 3(b), when a customer's hand 60 enters the product display area 12 in the image and picks up a product 10, the detection unit 104 detects that the customer has picked up the product 10.

[0069] 6(a), the detection unit 104 may detect that a product 10 has been taken and record the result in the detection result 200. The detection result 200 stores information indicating at least one of the following: image data of the product that has been detected as having been taken, identification information of the image (for example, a frame number), the time of photographing, identification information of the self-checkout terminal 50 that the customer was operating, and the name of the image data.

[0070] When the detection unit 104 detects that the customer has taken a product (YES in step S103), the processing unit 106 performs the following process (a) and process (b). In process (a), if the transaction process has not yet been completed (NO in step S105), information prompting the user to register the product is output to the output device (step S109). In process (b), if the transaction process has been completed (YES in step S105), information that can identify the customer is stored in storage device 120 (step S107).

[0071] In step S109, the processing unit 106 may display information such as "Please scan the product" on the touch panel 52 of the self-checkout terminal 50 or output the information as audio from the speaker of the self-checkout terminal 50. Then, the process may return to step S101 and continue monitoring whether the customer picks up another product 10.

[0072] In step S107, the processing unit 106 may add the customer-identifying information to the detection result 200 and record it in the storage device 120, as shown in FIG. 6(b), for example. Then, this process ends. The process may return to step S101 to prepare for the next customer.

[0073] As described above, according to this embodiment, the monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 operated by the customer, and the detection unit 104 processes the image to detect that the customer has taken a product 10 from the product display area 12. When the detection unit 104 detects that the customer has taken a product 10, the processing unit 106 performs the following processes (a) and (b). Process (a), if the transaction has not yet been completed, causes the output device to output information prompting the customer to register the product. Process (b), if the transaction has already been completed, causes the storage device 120 to store information that can identify the customer.

[0074] Specifically, when the detection unit 104 detects that a customer has picked up an item 10 located near the self-checkout terminal 50, if the transaction process at the self-checkout terminal 50 has not yet been completed, the processing unit 106 can display information on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the picked up item 10, or output audio from the speaker of the self-checkout terminal 50. Alternatively, after the detection unit 104 detects that a customer has picked up an item 10 located near the self-checkout terminal 50, the transaction process at the self-checkout terminal 50 is completed and the customer leaves, information that can identify the customer can be recorded in the detection result 200, allowing the store to manage the customer. This makes it possible to take measures to prevent transaction omissions.

[0075] As described above, monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0076] (Second embodiment) 7 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to the above-described embodiment, except that it has a configuration for determining whether the product picked up by the customer is included in the list of products read by the self-checkout terminal 50. The monitoring device 100 according to this embodiment further includes a determination unit 108 in addition to the configuration of the monitoring device 100 of FIG. 1. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 of FIG. 1, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0077] <Example of functional configuration> If the transaction process has not yet been completed, the determination unit 108 determines whether the product 10 picked up by the customer is included in the list of products read by the self-checkout terminal 50 in the subsequent product registration process. If it is determined that the product 10 is not included in the list, the processing unit 106 causes the output device to output information prompting further product registration processing.

[0078] The list of products read by the self-checkout terminal 50 is a list of POS data including at least the product names and prices of the products read by the self-checkout terminal 50 during the product registration process. Alternatively, the POS data may include the product codes of the products.

[0079] The determination unit 108 refers to the list of POS data and determines whether the product 10 detected by the detection unit 104 as having been taken by the customer is included. For example, the determination result may be stored in association with the detection result 200. The determination result includes at least one of information indicating whether the POS data of the product 10 was included in the list, and information indicating that the product is legitimate if the POS data is present, or information indicating that the product is fraudulent if the POS data is absent.

[0080] The method for identifying the product 10 picked up by the customer is exemplified below. The information of the product 10 identified here is, for example, at least one of the product name and product code of the product 10. (c1) The feature amount of the product 10 is registered in advance, and the image processing device 130 performs image recognition processing on the image of the product 10 to identify the product. (c2) Products to be displayed in each product display area 12 are registered in advance. By identifying the product display area 12 in the image, the product associated with the product display area 12 is identified as the product 10 picked up by the customer based on the position of the customer's hand 60.

[0081] Fig. 8 is a diagram showing an example of the data structure of the detection result 200. In this example, the detection result 200 in Fig. 6 is further linked with information indicating the presence or absence of POS data and information indicating whether the data is fraudulent or legitimate.

[0082] The "subsequent product registration process" refers to the product registration process performed by the self-checkout terminal 50 after it is detected that the customer has taken the product 10.

[0083] Here, the output device may be the same as in the first embodiment. The information prompting the customer to register the product may be the same as the information output after it is detected that the customer has taken the product 10, or may include information that further urges the customer to register the product. That is, when it is detected that the customer has taken the product 10, the processing unit 106 outputs information that initially calls for attention to prompt the customer to register the product, and then, if it is confirmed that the product 10 has not been scanned, outputs information to further emphasize the need to perform the registration process.

[0084] For example, if a customer picks up a product with the product name "CCC Gum," the information to be output may be, "There is a product that you have not yet scanned. Please scan the product (which may have the product name "CCC Gum").", "It seems that you forgot to scan the product (which may have the product name "CCC Gum"). Please scan it.", "CCC Gum is chargeable. Please scan it.", etc.

[0085] That is, the information to be output includes at least one of the following: a content urging the customer to register the product again, a content informing the customer that they have forgotten to register the product, and a content informing the customer that the product is subject to a fee.

[0086] In another example, the processing unit 106 outputs an alert to the output device when it is determined that the product 10 is not included in the list. This process by the processing unit 106 is also referred to as process (c). In this example, the output device includes at least one of the touch panel 52 of the self-checkout terminal 50, a speaker, signage, a monitoring device in the monitoring center, a manager terminal, and a store clerk terminal. For the customer, the processing unit 106 displays a warning message on the touch panel 52 of the self-checkout terminal 50 (for example, by flashing or highlighting an icon) or outputs an alert sound from the speaker to draw attention.

[0087] The alert may include information with a stronger message than information urging the customer to register the product. For example, the alert preferably includes information informing the customer that the picked-up product requires a fee and that it needs to be scanned, and instructing the customer to scan it, such as "Please scan the picked-up product," "There is a fee for the picked-up product. Please scan it," "You forgot to scan the picked-up product. Please scan it," or "The picked-up product requires a scan. Please scan it."

[0088] Furthermore, the processing unit 106 notifies the manager or store clerk by outputting a warning message including information informing the manager or store clerk that there is a customer who has not registered the product 10 to the device or terminal of the manager or store clerk. The processing unit 106 may send the message including information that can identify the customer to the terminal, or may send the message including information that can identify the self-checkout terminal 50 used by the customer to the terminal.

[0089] This configuration makes it possible to strongly encourage customers to register their products. In addition, since it is possible to notify a manager or store clerk, a nearby store clerk can directly call out to the customer and encourage them to register their products.

[0090] The following describes the operation of the monitoring device 100 according to the embodiment. Hereinafter, an operation example 1 will be described in which, when there is no POS data, information prompting product registration is output again, and an operation example 2 will be described in which an alert is output to the output device.

[0091] <Example 1> FIG. 9 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. This flow is executed after step S109 in Fig. 2. After the processing unit 106 outputs information prompting product registration in step S109, the determination unit 108 determines whether the product 10 picked up by the customer is included in the list of products read by the self-checkout terminal 50 in the subsequent product registration process (step S201). Specifically, the POS data list of the self-checkout terminal 50 is referenced to determine whether the POS data of the product 10 picked up by the customer is included in the list.

[0092] If the processing unit 106 determines that the product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output information prompting further product registration processing (step S203). For example, the processing unit 106 causes the touch panel 52 of the self-checkout terminal 50 to display a message such as "It seems you forgot to scan the product (which may be the product name "CCC Gum"). Please scan it." Then, the process returns to step S101 in FIG. 2.

[0093] If it is determined that the product 10 is included in the list (YES in step S201), the processing unit 106 ends this process. Alternatively, the processing unit 106 may return to step S101 in FIG. 2 and start monitoring whether another product 10 is taken.

[0094] In this way, if it is detected that a customer has taken a product 10 and the POS data list does not include the POS data of the product 10, information prompting the customer to register the product again can be output.

[0095] <Example 2> FIG. 10 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. This flow is also executed after step S109 in Fig. 2. Step S201 is the same as the flow in Fig. 9. Then, if it is determined that the product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output an alert (step S221). Then, the process returns to step S101 in Fig. 2.

[0096] Specifically, the processing unit 106 displays a warning message on the touch panel 52 of the self-checkout terminal 50 (for example, by flashing or highlighting an icon) and outputs an alert sound from the speaker to draw attention to the customer. Furthermore, the processing unit 106 notifies the manager or store clerk by outputting a warning message including information informing the manager or store clerk that there is a customer who has not registered the product 10 to the device or terminal of the manager or store clerk.

[0097] As described above, in the monitoring device 100 of this embodiment, before the transaction process is completed, the determination unit 108 determines whether the product 10 taken by the customer is included in the list of products read by the self-checkout terminal 50 in the subsequent product registration process, and if the processing unit 106 determines that the product 10 is not included in the list, it causes the output device to output information prompting further product registration processing.

[0098] According to this monitoring device 100, after detecting that a customer has taken a product 10 displayed near the self-checkout terminal 50, if the product 10 is not included in the list of registered products, a message urging the customer to register the product can be displayed on the touch panel 52 of the self-checkout terminal 50. In this way, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also effectively prevents customers from missing out on checking out products 10 in the product display area 12.

[0099] Furthermore, in the monitoring device 100 of this embodiment, the processing unit 106 outputs an alert to an output device when it is determined that the product 10 is not included in the list. According to this monitoring device 100, after detecting that a customer has taken a product 10 displayed near the self-checkout terminal 50, if the product 10 is not included in the list of registered products, an alert can be output not only to the self-checkout terminal 50 used by the customer, but also to in-store signage, a monitoring device in the monitoring center, a manager's terminal, and a store clerk's terminal. Thus, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also can notify a manager or store clerk that an item has been omitted from the transaction, enabling appropriate action to be taken, such as having the clerk directly speak to the customer and encourage them to register the product.

[0100] (Third embodiment) This embodiment is similar to any of the above-described embodiments, except that it has a configuration in which, when it is detected that a customer has picked up a product during the product registration process, information prompting the customer to register the product is output. The monitoring device 100 of this embodiment will be described below using FIG. 1 as having the same configuration as the monitoring device 100 of FIG. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0101] <Example of functional configuration> The detection unit 104 processes the image to detect that the customer has taken the product 10 during the product registration process. When the processing unit 106 detects that the customer has picked up the product 10, it causes the output device to output information prompting the customer to register the product 10 during the product registration process.

[0102] The processing performed by the processing unit 106 is the same as the processing (a) in the above embodiment. For example, in this embodiment, since the customer is currently performing a product registration operation on the self-checkout terminal 50, a product registration screen is displayed on the touch panel 52 of the self-checkout terminal 50. In this state, it is preferable to display information including at least one of the following: a message urging the customer to register the product 10 that the customer has picked up; a message informing the customer that the product 10 has not been registered; and a message informing the customer that the product 10 needs to be registered.

[0103] The criteria for determining whether or not a product registration process is in progress include the period during which the self-checkout terminal 50 is accepting product registration operations by a customer. In other words, the period during which a screen accepting product registration is displayed on the operation screen of the self-checkout terminal 50. The period during which a product registration process is in progress is the period up until the time when the checkout button to proceed to the checkout process is accepted on that screen.

[0104] <Example of operation> 11 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. This flow includes the same steps as the flow in FIG. 2, and further includes step S301 between step S103 and step S105 in the flow in FIG.

[0105] In this example, if the detection unit 104 detects in step S103 that the customer has picked up the product 10 (YES in step S103), the processing unit 106 determines whether the customer is currently in the process of registering the product at the self-checkout terminal 50 (step S301). If the customer is currently in the process of registering the product (YES in step S301), the process proceeds to step S109, where the processing unit 106 causes the output device to output information prompting the customer to register the product 10 (step S109). In other words, while a screen for registering the product is displayed on the touch panel 52 of the self-checkout terminal 50, the processing unit 106 causes the screen to display information including content prompting the customer to register the product 10 that the customer picked up.

[0106] On the other hand, if product registration processing is not in progress (NO in step S301), for example, if payment processing or settlement processing is in progress, the process proceeds to step S105. If the transaction processing has not yet been completed (NO in step S105), the process returns to step S301, unlike in FIG. 2. In other words, the process waits until the transaction processing is completed. Then, if the transaction processing has been completed (YES in step S105), information that can identify the customer is stored in storage device 120 (step S107).

[0107] As described above, in the monitoring device 100 of this embodiment, the detection unit 104 processes the image to detect that a customer has taken the product 10 during the product registration process, and when the processing unit 106 detects that the customer has taken the product 10, it causes the output device to output information prompting the customer to register the product 10 during the product registration process.

[0108] According to this monitoring device 100, while a customer is undergoing product registration processing, content encouraging the customer to register the product 10 that the customer has taken from the product display area 12 can be displayed on the operation screen of the self-checkout terminal 50 performing the product registration processing. As described above, the monitoring device 100 of this embodiment not only provides the same effects as the above-described embodiment, but also effectively prevents customers from forgetting to check out the products 10 in the product display area 12.

[0109] (Fourth embodiment) FIG. 12 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to any of the above-described embodiments, except that it has a configuration in which, after detecting that a customer has picked up a product, it is determined by image processing whether or not product registration processing for that product has been performed. The monitoring device 100 according to this embodiment further includes an identification unit 110 in addition to the configuration of the monitoring device 100 of FIG. 1. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 of FIG. 1, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments as long as no contradictions arise.

[0110] <Example of functional configuration> Before the transaction process is completed, the identification unit 110 processes the image after the customer picks up the product 10, and when the relative distance between the position of the product registration reading device (scanner 54) and the position of at least one of the customer's fingers and the product 10 meets a criterion, it identifies that the product registration process for the product 10 has been performed. The processing unit 106 causes the output device to output an alert on the condition that the product registration process has ended without the identification unit 110 identifying that the product 10 has been registered. This process is the same as the process (c) above.

[0111] Here, the method for determining whether or not a transaction has been completed before the transaction has been completed may be the same as the method described in the above embodiment.

[0112] The "end of product registration processing" occurs, for example, when the payment button on the product registration screen of the self-checkout terminal 50 is pressed.

[0113] The output device includes at least one of the touch panel 52 of the self-checkout terminal 50, a speaker of the self-checkout terminal 50, signage, a monitoring device in the monitoring center, a manager's terminal, and a store clerk's terminal.

[0114] Furthermore, after the transaction process is completed, the processing unit 106 may further cause an output device in the store to output information prompting the customer to perform registration processing for the product 10. This process is also referred to as process (d).

[0115] However, even before the transaction process is completed, the processing unit 106 may perform process (d) instead of or in addition to outputting the alert in process (c). The in-store output device may be, for example, at least one signage device installed in a store separate from the location of the self-checkout terminal 50. Alternatively, the in-store output device may include at least one speaker installed in the store.

[0116] The identification unit 110 causes the image processing device 130 to process the image after it is detected that the customer has taken the product 10. The image processing device 130 identifies the relative distance in the image between the position of the scanner 54 of the self-checkout terminal 50 and the position of at least one of the customer's fingers and the product 10. Criteria include whether the relative distance is within a predetermined range. For example, the predetermined range is a distance in the image that corresponds to a few centimeters to 10 centimeters in the real world.

[0117] Preferably, the criterion is that the relative distances in the image between the position of the scanner 54 of the self-checkout terminal 50, the position of the customer's fingers, and the position of the product 10 are within a predetermined range. In other words, the criterion is that the scanner 54 of the self-checkout terminal 50, the customer's fingers, and the product 10 are all included within the predetermined range.

[0118] When registering a product 10, a customer brings the barcode of the product 10 close to the scanner 54 to have it read. At this time, the product 10 and the scanner 54 are spaced a certain distance apart. The reading distance (DOF) of the scanner 54 varies depending on the type of scanner 54 and the type of barcode (e.g., JAN, CODE39, etc.). The reading distance DOF also has a minimum and maximum value. Therefore, it is preferable that the relative distance be within a range between at least the minimum and maximum values ​​of the reading distance DOF. However, because the maximum value of the reading distance DOF can be several tens of centimeters, using the maximum value as the upper limit of the range makes it impossible to determine whether the barcode of the product 10 has been read. Therefore, it is preferable not to use the maximum value of the reading distance DOF as the upper limit of the predetermined range. The predetermined range may also use the average distance at which the customer uses the scanner 54 to read the barcode of the product 10.

[0119] <How to set standards> Furthermore, since the type of barcode varies depending on the product, it is preferable to use a criterion that matches the type of barcode attached to the product 10 displayed in the product display area 12. When multiple types of products 10 with different types of barcodes attached are displayed in the product display area 12, the product display area 12 is divided according to the type of barcode attached to the product 10. Then, the type of barcode is registered for each divided product display area 12. Furthermore, a criterion is associated with each barcode type and registered. The identification unit 110 may identify the type of barcode corresponding to the product display area 12 of the product 10 picked up by the customer, and obtain a criterion (relative distance range) corresponding to the identified barcode.

[0120] In the second embodiment, whether or not the product registration process for the product 10 has been completed is determined based on whether or not the POS data for the product 10 is included in the list of products for which product registration has been processed at the self-checkout terminal 50. However, in the present embodiment, the POS data does not need to be used. Also, it is not necessary to identify the product name or product code of the product 10 picked up by the customer.

[0121] After it is detected that a customer has taken a product 10 from the product display area 12, it is determined whether or not an action of bringing the product 10 close to the scanner 54 of the self-checkout terminal 50 is detected, that is, when the relative distance between the position of the scanner 54 and the position of the product 10 or the customer's fingers becomes less than a threshold, it is determined that the product registration process for the product 10 has been performed.

[0122] There is a possibility that the same product 10 displayed in the product display area 12 may be displayed on another display shelf in the store. In such a case, the product displayed on the display shelf and the product 10 displayed in the product display area 12 cannot be distinguished from each other by product code. In other words, even if the POS data of a product 10 is included in the product list, that product is not necessarily the product 10 that was displayed in the product display area 12. It is possible that a product taken from a display shelf is scanned, but a product taken from the product display area 12 is forgotten to be scanned. In such a case, it is not possible to detect an omission of a product taken from the product display area 12.

[0123] On the other hand, according to the configuration of this embodiment, it is possible to determine whether the product 10 actually picked up by the customer has been registered by bringing it close to the scanner 54, and therefore it is possible to distinguish between products taken from the display shelf and products taken from the product display area 12 and determine whether the product has been registered.

[0124] <Example of operation> 13 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. This flow starts when the detection unit 104 detects that the customer has taken the product 10 in step S103 of FIG.

[0125] When the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), it sets a determination flag used to determine whether the customer's behavior is fraudulent to 1. The determination flag may be stored in the detection result 200 described above. Fig. 14 is a diagram showing an example of the data structure of the detection result 200 in which the determination flag is stored.

[0126] If the transaction has been completed (YES in step S105), the processing unit 106 proceeds to step S107 in FIG. 2 and executes process (b). That is, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107). On the other hand, if the transaction has not yet been completed (NO in step S105), the processing unit 106 executes process (a). That is, the processing unit 106 outputs information to the output device that prompts the user to register the product (step S109).

[0127] After step S109, if the transaction has not yet been completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0128] If it is determined that the criteria are met (YES in step S403), the identification unit 110 determines that the product registration process has been performed for the product 10, and resets the corresponding determination flag in the detection result 200 to 0 (step S409). Then, the process returns to step S101 in FIG. 2.

[0129] On the other hand, if it is determined that the criteria are not met (NO in step S403), the processing unit 106 determines whether the product registration process is completed (step S405). If it is determined that the product registration process is completed (YES in step S405), the processing unit 106 executes process (c) (step S407). That is, the processing unit 106 causes the output device to output an alert.

[0130] However, the above-mentioned process (d) may be performed in step S407. The processing unit 106 can perform at least one of the processes (c) and (d).

[0131] On the other hand, if it is not determined that the product registration process is completed (NO in step S405), the process returns to step S105. That is, during the product registration process, the processes in steps S105 to S403 are repeated to monitor whether the registration process of the product 10 is being performed.

[0132] Furthermore, processing unit 106 does not perform process (b) of step S107, process (a) of step S109, and process (c) of step S407 when the determination flag is 0. In a configuration using a determination flag, the processes of steps S101 to S103, steps S105 to S109, steps S403 to S409, and steps S405 to S407 may be executed independently.

[0133] The processing unit 106 constantly performs the process of detecting whether the customer has taken the product 10 by the detection unit 104 in the processes of steps S101 to S103, and after setting the determination flag in step S401, executes each process. At this time, after the determinations in steps S105 and S405, the processing unit 106 refers to the determination flag to determine whether or not to execute the process (b) in step S107, the process (a) in step S109, and the process (c) in step S407. In other words, if the determination flag is set to 1, fraud is suspected and the respective processes are executed. If the determination flag is set to 0, the respective processes are not executed.

[0134] As described above, in the monitoring device 100 of this embodiment, before the transaction is completed, the identification unit 110 processes the image after the customer picks up the product 10 and determines that the product registration process has been performed for that product 10 when the relative distance between the position of the product registration reading device (scanner 54) and the positions of at least one of the customer's fingers and the product 10 meets a criterion, and the processing unit 106 outputs an alert to the output device on the condition that the product registration process has ended without the identification unit 110 determining that the product 10 has been registered. Furthermore, after the transaction is completed, the processing unit 106 further outputs information to an output device in the store that prompts the customer to register the product 10.

[0135] According to this monitoring device 100, in order to determine whether the product 10 picked up by the customer has been registered, it is not necessary to identify the product code or product name of the product 10 picked up by the customer, nor is it necessary to use POS data of registered products. The monitoring device 100 can determine that the product registration process has been carried out by bringing the product 10 actually picked up by the customer close to the scanner 54. Furthermore, if the product registration process has not been carried out for the product 10 but has not yet been completed, an alert can be output when the product registration process has ended to notify the customer that the product 10 has not been registered.

[0136] In this way, the monitoring device 100 of this embodiment achieves the same effects as the above embodiment, and furthermore, it can distinguish between products that customers have taken from display shelves and products that they have taken from the product display area 12, and determine whether or not the products have been registered, thereby efficiently preventing customers from missing out on accounting for products 10 that they have taken from the product display area 12.

[0137] Furthermore, when the product registration process has finished before the transaction process is completed, an alert can be output to the customer standing in front of the self-checkout terminal 50 to notify them that the product 10 has not been registered, and when the transaction process has been completed and the customer has left the self-checkout terminal 50, a message can be displayed on signage or the like within the store urging the customer to register the product 10, so that the customer can be made aware of the missed transaction before leaving the store.

[0138] (Fifth embodiment) FIG. 15 is a functional block diagram showing an example of the logical configuration of a monitoring device 100 according to an embodiment. This embodiment is similar to any of the above-described embodiments, except that it has a configuration in which the position of a scanner used in product registration processing or a product, etc., within an image is identified and registered through image processing. The monitoring device 100 according to this embodiment further includes a registration unit 112 in addition to the configuration of the monitoring device 100 shown in FIG. 12. Here, the monitoring device 100 according to this embodiment will be described as being configured in combination with the monitoring device 100 shown in FIG. 12, but the configuration of this embodiment may also be combined with at least one of the configurations of the other embodiments as long as no contradictions arise.

[0139] <Example of functional configuration> The registration unit 112 processes the image to identify and pre-register the positions of at least one of the scanner 54, the product display area 12, and the product 10 within the image.

[0140] The detection unit 104 detects by image processing that a customer's hand 60 has entered a pre-registered product display area 12, and then detects that the product 10 has been taken by detecting that the product 10 is in the customer's hand 60. Furthermore, the identification unit 110 uses image processing to detect that the customer's hand 60 and the product 10 are present at the position of the pre-registered scanner 54, thereby identifying that the product 10 has been scanned.

[0141] <Example of operation> Fig. 16 is a flowchart showing an example of the operation of the monitoring device 100 according to the embodiment. Fig. 17 is a diagram for explaining a registration method performed by the registration unit 112. The following description will be given with reference to Figs. 16 and 17.

[0142] First, the image acquisition unit 102 acquires an image including the product display area 12 around the self-checkout terminal 50 (step S101). Then, the registration unit 112 displays the image 300 acquired in step S101 on the display device of the monitoring device 100, prompts the operator to mark the position of the product 10 to be monitored in the product display area 12 and the position of the scanner 54, and accepts the marking operation (step S601). Fig. 17(a) shows the state in which the product display area 12 has been marked.

[0143] The registration unit 112 causes the image processing device 130 to perform image processing based on the positions of the received markings 310, thereby identifying the product display area position 302 and the scan position 304 (step S603). Alternatively, the registration unit 112 may individually identify the position of each product 10 through image processing by the image processing device 130. As shown in FIG. 17(b), the frames indicating the identified product display area position 302 and scan position 304 are each shown with a different line type. For example, the line type may be different colors, different thicknesses, or different line types (e.g., solid lines, dashed lines, dashed lines, etc.), or the frame may be shaded with different types or colors of shading. The frames indicating the identified product display area position 302 and scan position 304 are superimposed on the image 300.

[0144] The specified result is displayed on the screen and confirmation is requested from the operator (step S605). If there is a change, the operator's operation to move or change the size of position 302 or position 304 is accepted, and the registration unit 112 changes position 302 or position 304 in accordance with the accepted content. Here, position 302 and position 304 are, for example, ranges indicated by coordinates.

[0145] Alternatively, the marking by the operator in step S601 may be omitted, and the positions of the product display area 12, the scanner 54, and the products 10 may be identified by image recognition processing by the image processing device 130 in step S603.

[0146] Then, the registration unit 112 registers the positions of at least one of the scanner 54, the product display area 12, and the product 10 in the image 300 as the position information 210 in the storage device 120 (step S607).

[0147] However, if the scanner 54 is a handheld scanner, even if its location is registered in advance, it will move, so it is preferable to identify the scanner's location through image recognition processing. Therefore, in the case of a handheld scanner, the configuration of the fourth embodiment described above determines whether the registration process for the product 10 has been completed based on whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion. Therefore, for handheld scanners, the advance location registration of this embodiment does not need to be performed.

[0148] However, if the location where the handheld scanner is placed is fixed, the location of the handheld scanner may be registered, since the process of determining the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 can be started after detecting whether the customer has placed their hand 60 on the handheld scanner.

[0149] In the position registration process in this embodiment, when the position of the product 10 or the position of the scanner 54 changes, the contents are updated by a store clerk or the like. Furthermore, by using an image captured by the camera 5 and processing the image in the image processing device 130, it may be possible to detect a change in the position of at least one of the product 10 and the scanner 54. When a change is detected, the registered position information may be updated. There are no particular limitations on the method for detecting a change in position using image processing, but it may be possible, for example, to detect the change by detecting the difference from the background image.

[0150] The position of the product display area 12 in the image may be determined in advance. As described above, the position can be determined by marking by the operator, or the position can be determined in advance by image processing. The detection unit 104 processes the image at this position to detect that the customer has taken the product 10 from the product display area 12.

[0151] In this way, if the position of the product display area 12 within the image is predetermined, it is only necessary to process the image at that position, which reduces the load on image processing and also makes it possible to improve image recognition accuracy, for example, by performing image recognition processing appropriate for the image of the product display area 12.

[0152] As described above, in the monitoring device 100 of this embodiment, the registration unit 112 processes the image to identify and pre-register the positions of at least one of the scanner 54, the product display area 12, and the product 10 within the image.

[0153] As described above, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also, since the positions of the product 10 to be detected and the scanner 54 are registered in advance, the area to be subjected to image recognition can be limited, thereby reducing the load on image processing and improving recognition accuracy.

[0154] (Sixth embodiment) In this embodiment, an example of a minimum configuration will be described, which has a configuration in which, when it is detected that a customer has picked up a product during the product registration process, information prompting the customer to register the product is output. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0155] <Example of functional configuration> The following explanation will be given with reference to FIG. The monitoring device 100 includes an image acquisition unit 102, a detection unit 104, and a processing unit 106. The image acquisition unit 102 is operated by a customer and acquires an image including the product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50) having a product registration function and a checkout function. The detection unit 104 detects that a customer has taken a product 10 from the product display area 12 by processing the image. When the detection unit 104 detects that the customer has taken the product 10, the processing unit 106 causes the output device to output information prompting the customer to register the product 10 during the product registration process.

[0156] The image acquisition unit 102 and the detection unit 104 are the same as those in the monitoring device 100 of the above-described embodiment.

[0157] <Example of operation> FIG. 18 is a flowchart showing an example of the operation of the monitoring device 100 of this embodiment. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). Then, the detection unit 104 determines whether or not the product registration process is in progress (step S301). Here, it can be determined that the product registration process is in progress while the operation screen for performing the product registration process is displayed on the self-checkout terminal 50.

[0158] Then, if the product registration process is in progress (YES in step S301), the detection unit 104 causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that the customer has taken the product 10 from the product display area 12 (step S103). If the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), the processing unit 106 causes the output device to output information prompting the customer to register the product 10 during the product registration process (step S109).

[0159] On the other hand, if the product registration process is not in progress (NO in step S301), for example, before entering the product registration process, during the settlement process, or during the payment process, the process returns to step S101. Also, if the detection unit 104 does not detect that the customer has taken the product 10 (NO in step S103), the process returns to step S101.

[0160] According to this monitoring device 100, the image acquisition unit 102 acquires an image including a product display area 12 within a predetermined range from the self-checkout terminal 50, the detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12, and when the detection unit 104 detects that a customer has taken a product 10, the processing unit 106 causes an output device to output information prompting the customer to register the product 10 during the product registration process.

[0161] Specifically, when a customer is in the process of registering a product at the self-checkout terminal 50 and the detection unit 104 detects that the customer has picked up a product 10 that is located near the self-checkout terminal 50, the processing unit 106 can display information on the touch panel 52 of the self-checkout terminal 50 that is in the process of registering the product, encouraging the customer to register the picked-up product 10, or output audio from the speaker of the self-checkout terminal 50.

[0162] In this way, the monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0163] Seventh embodiment In this embodiment, an example of a minimum configuration will be described, which has a configuration in which, after detecting that a customer has picked up a product, it is determined by image processing whether or not the product registration process for that product has been performed. However, the configuration of this embodiment may be combined with at least one of the configurations of the other embodiments to the extent that no contradiction occurs.

[0164] <Example of functional configuration> The following description will be given with reference to FIG. The monitoring device 100 includes an image acquisition unit 102 , a detection unit 104 , a processing unit 106 , and an identification unit 110 . The image acquisition unit 102 is operated by a customer and acquires an image including the product display area 12 within a predetermined range from a checkout device (self-checkout terminal 50) having a product registration function and a checkout function. The detection unit 104 detects that a customer has taken a product 10 from the product display area 12 by processing the image. If the transaction process has not yet been completed, the identification unit 110 processes the image after the customer picks up the product 10, and when the relative distance between the position of the reading means for product registration and the position of at least one of the customer's fingers and the product 10 satisfies a standard, identifies that the product registration process for the product 10 has been performed. The processing unit 106 causes the output device to output an alert on the condition that the product registration process has ended without the identification unit 110 identifying that the product 10 has been registered (processing (c)). <Example of operation> FIG. 19 is a flowchart showing an example of the operation of the monitoring device 100 of this embodiment. First, the image acquisition unit 102 acquires an image including the product display area 12 within a predetermined range from the self-checkout terminal 50 (step S101). Then, the detection unit 104 causes the image processing device 130 to process the image acquired by the image acquisition unit 102, thereby detecting that a customer has taken a product 10 from the product display area 12 (step S103).

[0165] If the detection unit 104 detects that the customer has taken the product 10 (YES in step S103), and if the transaction process has not yet been completed (NO in step S105), the identification unit 110 causes the image processing device 130 to process the image after the customer has taken the product 10, and determines whether the relative distance between the position of the reading means (scanner 54) for product registration and the position of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0166] If it is determined that the criteria are met (YES in step S403), the identification unit 110 determines that the product registration process for the product 10 has been completed, and returns to step S101. That is, the identification unit 110 continues to monitor whether the next product 10 is taken from the product display area 12 by the customer.

[0167] On the other hand, if it is determined that the criteria are not met (NO in step S403), the processing unit 106 determines whether the product registration process is completed (step S405). If it is determined that the product registration process is completed (YES in step S405), the processing unit 106 executes the above process (c) (step S407). That is, the processing unit 106 causes the output device to output an alert.

[0168] On the other hand, if it is not determined that the product registration process is complete (NO in step S405), the process returns to step S101. That is, the process continues to monitor whether the next product 10 is taken from the product display area 12 by the customer.

[0169] According to this monitoring device 100, the image acquisition unit 102 acquires an image including a product display area 12 within a predetermined range from the self-checkout terminal 50, the detection unit 104 processes the image to detect that a customer has taken a product 10 from the product display area 12, and the identification unit 110 processes the image after the customer has taken the product 10, if the transaction process has not yet been completed, and identifies that product registration processing has been performed for the product 10 when the relative distance between the position of the product registration reading means and the position of at least one of the customer's fingers and the product 10 meets a criterion, and the processing unit 106 outputs an alert to the output device on the condition that the product registration process has been completed without the identification unit 110 identifying that product registration processing has been performed for the product 10.

[0170] According to this monitoring device 100, in order to determine whether the product 10 picked up by the customer has been registered, it is not necessary to identify the product code or product name of the product 10 picked up by the customer, nor is it necessary to use POS data of registered products. The monitoring device 100 can determine that the product registration process has been carried out by bringing the product 10 actually picked up by the customer close to the scanner 54. Furthermore, if the product registration process has not been carried out for the product 10 but has not yet been completed, an alert can be output when the product registration process has ended to notify the customer that the product 10 has not been registered.

[0171] In this way, the monitoring device 100 can solve the problem of missing a certain product in the vicinity of a payment device that is operated by a customer and has product registration and payment functions.

[0172] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted. For example, when the detection unit 104 detects that a customer has taken a product 10, the processing unit 106 may output an instruction to register the product 10, while the processing unit 106 may prevent the display of the checkout button for proceeding to the checkout process from the operation screen that performs the product registration process on the self-checkout terminal 50, or may display the checkout button as inactive so that pressing the button is not accepted. [Example]

[0173] First, an example of the operation when a customer performs product registration processing and payment processing at the self-checkout terminal 50 will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of the procedure for product registration processing and payment processing at the self-checkout terminal 50.

[0174] First, a customer stands in front of the self-checkout terminal 50 (Step S1). Then, the customer places the product to be purchased or a shopping basket containing the product on the counter (Step S3). Then, the customer starts operating the self-checkout terminal 50 (Step S5). For example, at this time, a standby screen is displayed on the touch panel 52 of the self-checkout terminal 50. When the customer touches the touch panel 52, an initial screen for the product registration process is displayed.

[0175] The initial screen displays an operation screen for selecting whether or not a plastic bag is required, and the customer selects whether or not a plastic bag is required (Step S7). However, the processing of Step S7 may be omitted. After selecting the plastic bag, the screen transitions to the product registration processing screen, where the customer reads the product's barcode with scanner 54 (Step S9). The product whose barcode has been read is registered. The customer then places the scanned product into the plastic bag (or a shopping bag they brought with them) (Step S11).

[0176] Once all items have been scanned, the customer presses the checkout button (step S13). The screen then transitions to a payment method selection screen, where the customer selects a payment method (step S15). The customer pays using the selected payment method (step S17). The customer receives any change and a receipt (step S19). The customer then takes the items and leaves the store (step S21). The customer then exits the store (step S23).

[0177] When a customer performs the steps shown in FIG. 20 in front of the self-checkout terminal 50, specific examples of the output content and output timing to prevent impulse purchases from being overlooked will be explained for each of the above embodiments.

[0178] Example 1 21 is a diagram illustrating an example of operation of Example 1. In Example 1, a case will be described in which the monitoring device 100 performs operations according to the flow of FIG. 2 of the first embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1), when the customer performs the product registration process at the self-checkout terminal 50 (step S9), when the payment process is performed (step S13), when the customer makes the payment (step S17), and when the customer receives the change and receipt (step S19), if the detection unit 104 detects that the customer has taken the product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0179] After the customer receives the change and receipt (step S19), the processing unit 106 stores information that can identify the customer who has been detected by the detection unit 104 to have taken the product 10 in the storage device 120 (step S107).

[0180] In this way, in the first embodiment, it is possible to effectively prevent the omission of a transaction for a product 10 that is located near the self-checkout terminal 50. In addition, since it is possible to manage information on customers who leave without registering the product, it is possible to take measures to prevent the omission of a transaction.

[0181] Example 2 Fig. 22 is a diagram illustrating an example of operation of Example 2. In Example 2, a case will be described in which the monitoring device 100 performs an operation according to the flow of Fig. 9 of Example 1 of the second embodiment. From the time a customer stands in front of the self-checkout terminal 50 (step S1), until the customer performs the product registration process at the self-checkout terminal 50 (step S9), performs the settlement process (step S13), makes the payment (step S17), and receives the change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken the product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0182] On the other hand, if it is detected that the customer has taken product 10 from the product display area 12 before the transaction process is completed and it is determined that product 10 is not included in the list (NO in step S201), the processing unit 106 causes the output device to output information (message 410) prompting further product registration processing (step S203).

[0183] Also, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the product 10 has been taken and determines that the product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0184] 23 to 25 are diagrams illustrating the timing of output of message 400 and message 410. In the example of FIG. 23, a list screen 450 of registered products in the product registration process is displayed on touch panel 52 of self-checkout terminal 50. The customer is then registering two products (step S9). During this product registration process, it is detected that the customer has taken product 10 (product C) from product display area 12 (step S103). Then, processing unit 106 causes message 400, including information prompting the customer to register product C, to be output on registered product list screen 450 displayed on touch panel 52 of self-checkout terminal 50.

[0185] Then, when the customer presses the checkout button to proceed to the checkout process (step S13), the product registration process is completed, and the processing unit 106 then outputs a message 410 containing information urging the customer to register product C on the registered product list screen 450 displayed on the touch panel 52 of the self-checkout terminal 50 before proceeding to the checkout process. This message 410 is more forceful than the message 400 in urging the customer to scan product 10.

[0186] 24 shows examples of messages 402 and 412 that are output by processing unit 106 when detection unit 104 detects that a customer has taken a shopping bag before the transaction is completed. The example of Fig. 25 shows examples of messages 404 and 414 that are output by processing unit 106 when detection unit 104 detects that a customer has taken a coffee cup for self-service coffee before the transaction is completed.

[0187] In this way, in the second embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, it is checked whether the POS data of the product 10 is included in the list of registered products, and if the product 10 is not included in the list, information urging the customer to register the product 10 is output again, making it possible to more effectively prevent the customer from missing out on the transaction of a product 10 that is located near the self-checkout terminal 50. In addition, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent the customer from missing out on the transaction.

[0188] Example 3 Fig. 26 is a diagram showing an operation example of Example 3. In Example 3, a case will be described in which the monitoring device 100 performs an operation according to the flow of Fig. 10 of Operation Example 2 of the second embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0189] On the other hand, if it is detected that a customer has taken a product 10 from the product display area 12 before the transaction is completed and it is determined that the product 10 is not included in the list (NO in step S201), the processing unit 106 outputs an alert to the output device (step S221). For example, the processing unit 106 outputs a message to the store clerk terminal 70 notifying the customer that there is a customer who has not registered the product 10 in the product display area 12. Alternatively, the processing unit 106 may display information on the touch panel 52 of the self-checkout terminal 50 that strongly urges the customer to register the taken product 10, or may output an alert sound or message from the speaker of the self-checkout terminal 50.

[0190] Also, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the product 10 has been taken and determines that the product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0191] In this way, in the third embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, it is checked whether the POS data of the product 10 is included in the registered product list, and if the product 10 is not included in the list, an alert is output to the output device, making it possible to more effectively prevent products 10 left around the self-checkout terminal 50 from being overlooked. In addition, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent overlooking of products.

[0192] Example 4 Fig. 27 is a diagram illustrating an example of operation of Example 4. In Example 4, a case will be described in which the monitoring device 100 performs an operation according to the flow of Fig. 13 of the fourth embodiment. Between the time when a customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives their change and receipt (step S19), that is, before the transaction process is completed, if the detection unit 104 detects that the customer has taken product 10 from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0193] Furthermore, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the customer has taken the product 10 and determines that the product 10 is not included in the list, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107). Furthermore, the processing unit 106 outputs an alert (message 430) to the output device. For example, the processing unit 106 displays the message 430 on the touch panel 52 of the self-checkout terminal 50 and outputs an alarm sound and voice from the speaker of the self-checkout terminal 50. This message 430 is one level stronger than the message 400 and warns the customer to scan the product 10.

[0194] In this way, in the fourth embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, information prompting the customer to register the product 10 is output, and if the customer then completes the transaction and leaves the self-checkout terminal 50, an alert can be output on signage or the like in the store, rather than on the touch panel 52 or speaker of the self-checkout terminal 50. In this way, the warning to register the product can be continued even after the customer has left the self-checkout terminal 50, so it is possible to more effectively prevent the customer from missing out on the transaction of products 10 that are located near the self-checkout terminal 50. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent missing out on the transaction.

[0195] Example 5 Fig. 28 is a diagram illustrating an example of operation of Example 5. In Example 5, a case will be described in which the monitoring device 100 performs operations according to the flow of Fig. 11 of the third embodiment. Between the time when the customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives the change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs a process to detect that the customer has taken the product 10 from the product display area 12 (step S103).

[0196] Then, between the time when the customer starts the product registration process (step S5) and the time when the customer presses the checkout button to move on to the checkout process (step S13), that is, during the product registration process, when the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50, urging the customer to register the product 10 (step S109).

[0197] Meanwhile, after the customer receives the change and receipt (step S19), the processing unit 106 stores information capable of identifying the customer who was detected by the detection unit 104 to have taken the product 10 in the storage device 120 (step S107).

[0198] As described above, in the fifth embodiment, when it is detected that a customer has taken a product 10 from the product display area 12, during the product registration process, information urging the customer to register the product 10 can be displayed on the touch panel 52 of the self-checkout terminal 50 performing the product registration process, thereby more effectively preventing the customer from missing out on the transaction for the product 10 that is located near the self-checkout terminal 50. Furthermore, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent the customer from missing out on the transaction.

[0199] Example 6 Fig. 29 is a diagram illustrating an example of operation of Example 6. In Example 6, a case will be described in which the monitoring device 100 performs an operation according to the flow of Fig. 13 of the fourth embodiment. Between the time when the customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives the change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs a process to detect that the customer has taken the product 10 from the product display area 12 (step S103).

[0200] Then, when the detection unit 104 detects that the product 10 has been taken from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) urging the customer to register the product 10 on the touch panel 52 of the self-checkout terminal 50 (step S109) until the product registration process is completed, that is, until the checkout button is pressed to move on to the checkout process (step S13).

[0201] Then, before the customer receives the change and receipt (step S19), i.e., before the transaction is completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0202] If it is determined that the criteria are not met (NO in step S403) and the product registration process is completed, that is, after the checkout button is pressed (step S13), the processing unit 106 outputs an alert to the output device (step S407). For example, the processing unit 106 displays a message 430 on the touch panel 52 of the self-checkout terminal 50 and outputs an alarm sound and voice from the speaker of the self-checkout terminal 50. This message 430 is one level stronger than the message 400 and warns the customer to scan the product 10.

[0203] Furthermore, after the customer receives the change and receipt (step S19), if the detection unit 104 detects that the product 10 has been taken and determines that the product 10 has not been scanned, the processing unit 106 stores information that can identify the customer in the storage device 120 (step S107).

[0204] In this way, in the sixth embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, information prompting the customer to register the product 10 is output. If the customer does not scan the product 10 before the product registration process is completed, an alert can be output after the product registration process is completed. The alert can include alerts to the customer and alerts to store clerks or managers. In addition, since it is possible to manage information on customers who leave without registering the product, measures can be taken to prevent customers from missing out on transactions.

[0205] Example 7 Fig. 30 is a diagram illustrating an example of operation of Example 7. In Example 7, a case will be described in which the monitoring device 100 performs an operation according to the flow of Fig. 18 of the sixth embodiment. After the customer starts the product registration process (step S5) and before pressing the checkout button to proceed to the checkout process (step S13), that is, during the product registration process, the detection unit 104 performs a process to detect that the customer has taken a product 10 from the product display area 12 (step S103).

[0206] During the product registration process, when the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0207] In this way, in Example 7, if it is detected that a customer has taken product 10 from the product display area 12 during the product registration process, information urging the customer to register product 10 can be displayed on the touch panel 52 of the self-checkout terminal 50 performing the product registration process, thereby making it possible to more effectively prevent products 10 stored near the self-checkout terminal 50 from being overlooked.

[0208] Example 8 Fig. 31 is a diagram showing an example of operation of Example 8. In Example 8, a case will be described in which the monitoring device 100 performs operations in accordance with the flow of Fig. 19 of the seventh embodiment. Between the time when the customer stands in front of the self-checkout terminal 50 (step S1) and the time when the customer receives the change and receipt (step S19), that is, before the transaction process is completed, the detection unit 104 performs a process to detect that the customer has taken the product 10 from the product display area 12 (step S103).

[0209] When the detection unit 104 detects that the product 10 has been removed from the product display area 12 (YES in step S103), the processing unit 106 displays information (message 400) on the touch panel 52 of the self-checkout terminal 50 prompting the customer to register the product 10 (step S109).

[0210] Then, before the customer receives the change and receipt (step S19), i.e., before the transaction is completed, the identification unit 110 processes the image after the customer picks up the product 10 to determine whether the relative distance between the position of the scanner 54 and the positions of at least one of the customer's fingers and the product 10 satisfies a criterion (step S403).

[0211] If it is determined that the criteria are not met (NO in step S403) and the product registration process is completed, that is, after the checkout button is pressed (step S13), the processing unit 106 outputs an alert to the output device (step S407). For example, the processing unit 106 displays a message 430 on the touch panel 52 of the self-checkout terminal 50 and outputs an alarm sound and voice from the speaker of the self-checkout terminal 50. This message 430 is one level stronger than the message 400 and warns the customer to scan the product 10.

[0212] In this way, in the eighth embodiment, after it is detected that a customer has taken a product 10 from the product display area 12, if the product 10 is not scanned by the time the product registration process is completed after outputting information prompting the customer to register the product 10, an alert can be output after the product registration process is completed. The alert can include alerts to the customer and alerts to store clerks or managers.

[0213] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0214] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In the present invention, when information about users (customers) is acquired and used, it shall be done lawfully.

[0215] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having a product registration function and a payment settlement function; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detection means detects that the customer has taken the product, If the transaction has not yet been completed, output information prompting the user to register the product; A monitoring device comprising: a processing means for storing information that can identify the customer in a storage means after the transaction process has been completed. 2. In the monitoring device described in 1., A monitoring device in which the information that can identify the customer includes at least one of identification information of the electronic payment medium used by the customer for the transaction, an image including the customer, and human features extracted from the image. 3. In the monitoring device according to 1. or 2., If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; The processing means, when it is determined that the product is not included in the list, causes the output means to output information prompting further product registration processing. 4. In the monitoring device according to any one of 1. to 3., If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product selected by the customer is included in a list of products read by the settlement device in a subsequent product registration process; The processing means causes the output means to output an alert when it is determined that the product is not included in the list. 5. In the monitoring device according to any one of 1. to 4., the detection means detects that the customer has picked up a product during product registration processing by processing the image; When the processing means detects that the customer has picked up the product, the processing means causes the output means to output information prompting the customer to register the product during the product registration process. 6. In the monitoring device according to any one of 1. to 5., If the transaction has not yet been completed, the system further comprises an identification means for identifying that the product registration process has been completed for the product by processing the image after the customer has picked up the product and, when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a standard, The processing means The monitoring device causes the output means to output an alert on the condition that the product registration process has ended without the identification means identifying that the product registration process has been performed for the product. 7. In the monitoring device according to 6., The processing means further causes an output means in the store to output information prompting the customer to register the product after the transaction process has been completed. 8. In the monitoring device according to 6. or 7., The monitoring device further comprises a registration means for specifying and pre-registering the position of the reading means within the image by processing the image. 9. In the monitoring device according to any one of 6. to 8., The monitoring device includes a registration means for specifying and pre-registering the position of at least one of the product display area and the product in the image by processing the image. 10. The monitoring device according to any one of 6. to 9. The position of the product display area in the image is predetermined, The detection means detects that the customer has taken the product from the product display area by processing the image of this position.

[0216] 11. Monitoring equipment; a payment device that is operated by a customer and has a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the checkout device; The monitoring device image acquisition means for acquiring the image; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detection means detects that the customer has taken the product, If the transaction has not yet been completed, output information prompting the user to register the product; A monitoring system having a processing means that, after the transaction process is completed, stores information that can identify the customer in a storage means. 12. In the monitoring system according to 11, A monitoring system in which the information that can identify the customer includes at least one of identification information of the electronic payment medium used by the customer for the transaction, an image including the customer, and human features extracted from the image. 13. In the monitoring system according to 11. or 12., The monitoring device If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; A monitoring system in which, when it is determined that the product is not included in the list, the processing means of the monitoring device causes the output means to output information prompting further product registration processing. 14. In the monitoring system according to any one of 11 to 13, The monitoring device If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product selected by the customer is included in a list of products read by the settlement device in a subsequent product registration process; A monitoring system, wherein the processing means of the monitoring device causes the output means to output an alert when it is determined that the product is not included in the list. 15. In the monitoring system according to any one of items 11 to 14, the detection means of the monitoring device processes the image to detect that the customer has taken the product during the product registration process; A monitoring system in which, when the processing means of the monitoring device detects that the customer has taken the product, it causes the output means to output information prompting the customer to register the product during the product registration process. 16. In the monitoring system according to any one of 11 to 15, The monitoring device If the transaction has not yet been completed, the system further comprises an identification means for identifying that the product registration process has been completed for the product by processing the image after the customer has picked up the product and, when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a standard, The processing means of the monitoring device The monitoring system causes the output means to output an alert on the condition that the product registration process has ended without the identification means identifying that the product registration process has been performed for the product. 17. In the monitoring system according to 16., The monitoring system further comprises: after the transaction process is completed, the processing means of the monitoring device causes an output means in the store to output information prompting the customer to register the product. 18. In the monitoring system according to 16. or 17., The monitoring device A surveillance system comprising a registration means for specifying and pre-registering the position of the reading means within the image by processing the image. 19. In the monitoring system according to any one of 16. to 18., The monitoring device A surveillance system comprising a registration means for specifying and pre-registering the positions of at least one of the product display area and the products in the image by processing the image. 20. In the monitoring system according to any one of 16. to 19., The position of the product display area in the image is predetermined, The detection means of the monitoring device detects that the customer has taken the product from the product display area by processing the image of this position.

[0217] 21. One or more computers acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; When it is detected that the customer has taken the product, If the transaction has not yet been completed, output information prompting the user to register the product; A monitoring method in which, after the transaction process is completed, information that can identify the customer is stored in a storage means. 22. In the monitoring method according to 21, A monitoring method in which the information that can identify the customer includes at least one of identification information of the electronic payment medium used by the customer in the transaction, an image including the customer, and human features extracted from the image. 23. In the monitoring method according to 21. or 22., the one or more computers: If the transaction has not yet been completed, in a subsequent product registration process, it is determined whether the product picked up by the customer is included in a list of products read by the settlement device; If it is determined that the product is not included in the list, the monitoring method causes the output means to output information prompting further product registration processing. 24. In the monitoring method according to any one of 21. to 23., the one or more computers: If the transaction has not yet been completed, in a subsequent product registration process, it is determined whether the product selected by the customer is included in a list of products read by the payment device; a monitoring method for causing the output means to output an alert when it is determined that the product is not included in the list; 25. In the monitoring method according to any one of 21. to 24., the one or more computers: processing the image to detect that the customer has taken an item during the item registration process; When it is detected that the customer has taken the product, the monitoring method causes the output means to output information prompting the customer to register the product during the product registration process. 26. In the monitoring method according to any one of 21. to 25., the one or more computers: If the transaction has not yet been completed, after the customer has picked up the product, the image is processed, and when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a criterion, it is determined that the product registration process for the product has been completed; A monitoring method in which an alert is output by the output means on condition that the product registration process is completed without the identification means identifying that the product registration process has been performed for the product. In the monitoring method described in 27.26., the one or more computers: If the transaction process has been completed, the monitoring method further causes an output means in the store to output information prompting the customer to register the product. 28. In the monitoring method according to 26. or 27., the one or more computers: The monitoring method further comprises processing the image to identify and pre-register the position of the reading means within the image. 29. In the monitoring method according to any one of 26. to 28., the one or more computers: A monitoring method comprising processing the image to identify and pre-register the position of at least one of the product display area and the product within the image. 30. In the monitoring method according to any one of 26. to 29., The position of the product display area in the image is predetermined, the one or more computers: The method detects when the customer removes the product from the product display area by processing the image at this position.

[0218] 31. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; When it is detected in the detecting step that the customer has taken the product, If the checkout process has not yet been completed, outputting information to prompt the user to register the product to the output means; A computer-readable recording medium that stores a program for executing a procedure for storing information that can identify the customer in a storage means after the transaction process is completed. 32. The recording medium according to 31, The information that can identify the customer includes at least one of the following: identification information of the electronic payment medium used by the customer in the accounting process; an image that includes the customer; and human features extracted from the image. A computer-readable recording medium on which a program is recorded. 33. In the recording medium according to 31. or 32., If the transaction has not yet been completed, a step of determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedure: if it is determined that the product is not included in the list, output information to the output means prompting further product registration processing. 34. The recording medium according to any one of 31 to 33, If the transaction has not yet been completed, a step of determining whether the product selected by the customer is included in a list of products read by the settlement device in a subsequent product registration process; A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure for causing the output means to output an alert when it is determined that the product is not included in the list. 35. The recording medium according to any one of 31 to 34, detecting that the customer has taken an item during an item registration process by processing the image; A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure in which, when it is detected that the customer has picked up the product, the output means outputs information prompting the registration of the product during the product registration process. 36. The recording medium according to any one of 31. to 35. a step of determining, in the case where the transaction has not yet been completed, that the product has been registered when the relative distance between the position of a reading means for product registration and the positions of at least one of the customer's fingers and the product satisfies a criterion by processing the image after the customer has picked up the product; A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure for outputting an alert to the output means on the condition that the product registration process for the product has been completed without the product registration process being identified as having been completed by the identification procedure. 37. The recording medium according to 36., A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following procedure: after the transaction process is completed, outputting information prompting the user to register the product to an output means within the store. 38. In the recording medium according to 36. or 37., A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure for specifying and pre-registering the position of the reading means within the image by processing the image. 39. The recording medium according to any one of 36. to 38. A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure for processing the image to identify and pre-register the position of at least one of the product display area and the product within the image. 40. The recording medium according to any one of 36 to 39, The position of the product display area in the image is predetermined, A computer-readable recording medium having recorded thereon a program for causing a computer to execute a procedure for detecting that the customer has taken the product from the product display area by processing the image at this position.

[0219] 41. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; When it is detected in the detecting step that the customer has taken the product, If the checkout process has not yet been completed, outputting information to prompt the user to register the product to the output means; A program for executing a procedure for storing information that can identify the customer in a storage means after the transaction process is completed. 42. In the program described in 41., A program in which the information that can identify the customer includes at least one of the following: identification information of the electronic payment medium used by the customer for the accounting process, an image containing the customer, and human features extracted from the image. 43. In the program described in 41. or 42., If the transaction has not yet been completed, a step of determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; a program for causing a computer to execute a procedure for causing the output means to output information prompting further product registration processing when it is determined that the product is not included in the list; 44. In the program according to any one of 41. to 43., If the transaction has not yet been completed, a step of determining whether the product selected by the customer is included in a list of products read by the settlement device in a subsequent product registration process; a program for causing a computer to execute a procedure for causing the output means to output an alert when it is determined that the product is not included in the list. 45. In the program according to any one of 41. to 44., detecting that the customer has taken an item during an item registration process by processing the image; a program for causing a computer to execute a procedure for causing the output means to output information prompting the registration of the product during the product registration process when it is detected that the customer has picked up the product; 46. ​​In the program according to any one of 41. to 45., a step of determining, in the case where the transaction has not yet been completed, that the product has been registered when the relative distance between the position of a reading means for product registration and the positions of at least one of the customer's fingers and the product satisfies a criterion by processing the image after the customer has picked up the product; A program for causing a computer to execute a procedure for outputting an alert to the output means on the condition that the product registration process for the product has ended without being identified as having been performed by the identification procedure. 47. In the program described in 46., The program causes a computer to execute a procedure for outputting, after the transaction process has been completed, information prompting the customer to register the product to an output means within the store. 48. In the program described in 46. or 47., A program for causing a computer to execute a procedure for specifying and pre-registering the position of the reading means within the image by processing the image. 49. In the program according to any one of 46. to 48., A program for causing a computer to execute a procedure for specifying and pre-registering the positions of at least one of the product display area and the product within the image by processing the image. 50. In the program according to any one of 46. to 49., The position of the product display area in the image is predetermined, A program for causing a computer to execute a procedure for detecting that the customer has taken the product from the product display area by processing the image at this position.

[0220] 51. An image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having product registration and payment functions; a detection means for detecting whether a customer has taken a product from the product display area during product registration processing by processing the image; a processing means for causing an output means to output information prompting the customer to register the product during the product registration process when it is detected that the customer has taken the product. 52. In the monitoring device according to 51., A monitoring device in which the information that can identify the customer includes at least one of identification information of the electronic payment medium used by the customer for the transaction, an image including the customer, and human features extracted from the image. 53. In the monitoring device according to 51. or 52., If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; The processing means, when it is determined that the product is not included in the list, causes the output means to output information prompting further product registration processing. 54. In the monitoring device according to any one of 51. to 53., If the transaction has not yet been completed, the system further includes a determination unit that determines whether the product selected by the customer is included in a list of products read by the settlement device during a subsequent product registration process; The processing means causes the output means to output an alert when it is determined that the product is not included in the list. 55. In the monitoring device according to any one of 51. to 54., If the transaction has not yet been completed, the system further comprises an identification means for identifying that the product registration process has been completed for the product by processing the image after the customer has picked up the product and, when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a standard, The processing means The monitoring device causes the output means to output an alert on the condition that the product registration process has ended without the identification means identifying that the product registration process has been performed for the product. 56. In the monitoring device according to 55., The processing means further causes an output means in the store to output information prompting the customer to register the product after the transaction process has been completed. 57. In the monitoring device according to 55. or 56., The monitoring device further comprises a registration means for specifying and pre-registering the position of the reading means within the image by processing the image. 58. In the monitoring device according to any one of 55. to 57., The monitoring device includes a registration means for specifying and pre-registering the position of at least one of the product display area and the product in the image by processing the image. 59. In the monitoring device according to any one of 55. to 58., The position of the product display area in the image is predetermined, The detection means detects that the customer has taken the product from the product display area by processing the image of this position.

[0221] 60. Surveillance equipment and a payment device that is operated by a customer and has a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the checkout device; The monitoring device an image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having a product registration and payment function; a detection means for detecting whether a customer has taken a product from the product display area during product registration processing by processing the image; a processing means for causing an output means to output information prompting the customer to register the product during the product registration process when it is detected that the customer has taken the product. 61. One or more computers acquiring an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration and payment function; processing the image to detect when a customer removes an item from the product display area during a product registration process; When it is detected that the customer has taken the product, the monitoring method causes an output means to output information prompting the customer to register the product during the product registration process. 62. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration and checkout function; processing the image to detect when a customer removes an item from the product display area during a product registration process; a program for executing a procedure for causing an output means to output information prompting the customer to register the product during the product registration process when it is detected that the customer has picked up the product; 63. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration and checkout function; processing the image to detect when a customer removes an item from the product display area during a product registration process; A computer-readable recording medium having recorded thereon a program for executing a procedure in which, when it is detected that the customer has taken the product, an output means is outputted with information prompting the customer to register the product during the product registration process.

[0222] 64. An image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having product registration and payment functions; a detection means for detecting when a customer has taken an item from the product display area by processing the image; an identification means for identifying that product registration processing has been performed for the product when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a standard by processing the image after the customer has picked up the product before the transaction process is completed; A monitoring device comprising: a processing means for causing an output means to output an alert on the condition that the product registration process has ended without the identification means identifying that the product registration process has been performed for the product. 65. In the monitoring device according to 64., A monitoring device in which the information that can identify the customer includes at least one of identification information of the electronic payment medium used by the customer for the transaction, an image including the customer, and human features extracted from the image. 66. In the monitoring device according to 64. or 65., If the transaction process has not yet been completed, the system further comprises a determination means for determining whether the product picked up by the customer is included in a list of products read by the settlement device in a subsequent product registration process; The processing means, when it is determined that the product is not included in the list, causes the output means to output information prompting further product registration processing. 67. In the monitoring device according to any one of items 64 to 66, If the transaction has not yet been completed, the system further includes a determination unit that determines whether the product selected by the customer is included in a list of products read by the settlement device during a subsequent product registration process; The processing means causes the output means to output an alert when it is determined that the product is not included in the list. 68. In the monitoring device according to any one of items 64 to 67, the detection means detects that the customer has picked up a product during product registration processing by processing the image; When the processing means detects that the customer has picked up the product, the processing means causes the output means to output information prompting the customer to register the product during the product registration process. 69. In the monitoring device according to any one of 64. to 68., The processing means further causes an output means in the store to output information prompting the customer to register the product after the transaction process has been completed. 70. In the monitoring device according to any one of paragraphs 64 to 69, The monitoring device further comprises a registration means for specifying and pre-registering the position of the reading means within the image by processing the image. 71. In the monitoring device according to any one of items 64 to 70, The monitoring device includes a registration means for specifying and pre-registering the position of at least one of the product display area and the product in the image by processing the image. 72. In the monitoring device according to any one of 64. to 71., The position of the product display area in the image is predetermined, The detection means detects that the customer has taken the product from the product display area by processing the image of this position.

[0223] 73. Surveillance equipment; a payment device that is operated by a customer and has a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the checkout device; The monitoring device an image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a payment device having a product registration and payment function; a detection means for detecting when a customer has taken an item from the product display area by processing the image; an identification means for identifying that product registration processing has been performed for the product when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a standard by processing the image after the customer has picked up the product before the transaction process is completed; A monitoring system having a processing means for causing an output means to output an alert on the condition that the product registration process has ended without the identification means identifying that the product registration process has been performed for the product. 74. One or more computers acquiring an image including a product display area within a predetermined range from a payment device operated by a customer and having a product registration and payment function; processing the image to detect when a customer has removed an item from the merchandise display area; After the customer has picked up the product before the transaction is completed, the image is processed, and when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a criterion, it is determined that product registration processing for the product has been completed; The monitoring method includes causing an output means to output an alert on condition that the product registration process has ended without it being determined that the product registration process has been performed for the product. 75. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration and checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; a step of processing the image after the customer has picked up the product before the transaction is completed, and determining that product registration processing for the product has been completed when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a criterion; A program for executing a procedure for outputting an alert to an output means on the condition that the product registration process for the product has been completed without being identified as having been performed by the identification procedure. 76. To the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration and checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; a step of processing the image after the customer has picked up the product before the transaction is completed, and determining that product registration processing for the product has been completed when the relative distance between the position of the product registration reading means and the positions of at least one of the customer's fingers and the product satisfies a criterion; A computer-readable recording medium having recorded thereon a program for executing a procedure for outputting an alert to an output means on the condition that the product registration process for the product has been completed without being identified as having been performed by the identification procedure. [Explanation of symbols]

[0224] 1. Surveillance System 3. Communication Network 5. Camera 10 items 12 Product display area 20 Plastic Bags 30 Coffee Cups 50 Self-checkout terminals 52 Touch Panel 54 Scanner 60 moves 70 Store clerk terminal 100 Monitoring equipment 102 Image acquisition unit 104 Detector 106 Processing section 108 Judgment section 110 Specific section 112 Registration Department 120 Storage device 130 Image processing device 200 detection results 210 Location information 1000 computers 1010 Bus 1020 processor 1030 memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface

Claims

1. an image acquisition means that is operated by a customer and acquires an image including a product display area within a predetermined range from a checkout device having a product registration function and a checkout function; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detecting means detects that the customer has picked up the product, the processing unit detects that the customer has completed the transaction; outputting information prompting the user to register the product until completion of the transaction is detected; a processing means for storing information capable of identifying the customer in a storage means after the completion of the transaction process is detected.

2. 2. The monitoring device according to claim 1, The processing means determines that the completion of the transaction process has been detected based on at least one of the following (a1) to (a3): (a1) detecting that the customer who was in front of the payment device has left by processing the image; (a2) detecting that the receipt or change has been received by the customer at the settlement device; (a3) by processing the image, after it is detected that the customer has left the payment device, until it is detected that the customer has left the store through an exit.

3. 3. The monitoring device according to claim 1, The processing means determines that the completion of the transaction process has not yet been detected based on at least one of the following (b1) to (b5): (b1) detecting that the customer is standing in front of the payment device by processing the image; (b2) After the settlement device accepts a transaction start operation from the customer. (b3) While the settlement device is accepting a product registration operation by the customer (b4) while the settlement device is accepting settlement processing operations by the customer (b5) while the settlement device detects that the customer has not received a receipt or change;

4. The monitoring device according to any one of claims 1 to 3, The information prompting the registration process for the product includes information indicating at least one of the following (c1) to (c4): (c1) the name of the product that the customer was detected to have taken; (c2) The product detected as having been picked up by the customer has a content emphasizing that the customer has just picked up the product. (c3) Content informing the customer that registration processing has not been performed for the product detected as having been taken by the customer. (c4) Content informing the customer that registration processing is required for the product detected as having been taken by the customer.

5. 5. The monitoring device according to claim 1, The information prompting the registration process for the product includes an operation screen that accepts an instruction input operation related to the product that the customer has detected as having taken, the monitoring device.

6. 6. The monitoring device according to claim 1, The customer identifiable information includes identification information of the electronic payment medium used by the customer in the transaction, associated with an image detected of the customer taking the item.

7. A monitoring device; a payment device that is operated by a customer and has a product registration function and a payment settlement function; a camera that generates an image including a product display area within a predetermined range from the checkout device; The monitoring device image acquisition means for acquiring the image; a detection means for detecting when a customer has taken an item from the product display area by processing the image; When the detecting means detects that the customer has picked up the product, the processing unit detects that the customer has completed the transaction; outputting information prompting the user to register the product until completion of the transaction is detected; a processing means for storing information capable of identifying the customer in a storage means after the completion of the transaction is detected.

8. One or more computers acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; When it is detected that the customer has picked up the product, a process is performed to detect the completion of the customer's checkout process; outputting information prompting the user to register the product until completion of the transaction is detected; After the completion of the transaction is detected, information that can identify the customer is stored in a storage means.

9. On the computer, a step of acquiring an image including a product display area within a predetermined range from a checkout device operated by a customer and having a product registration function and a checkout function; processing the image to detect when a customer has removed an item from the merchandise display area; In the detecting step, when it is detected that the customer has picked up the product, a process is performed to detect completion of the customer's checkout process; outputting information prompting the user to register the product until completion of the transaction is detected; a program for executing a procedure for storing information that can identify the customer in a storage means after the completion of the transaction process is detected;

Images