Information processing device and information processing system

Abstract

To inexpensively and easily notify a user of payment omission.SOLUTION: The information processing device includes a detection unit, a tracking unit, a linking unit, and a notification unit. The detection unit detects whether or not the operation related to the settlement is completed for each of the plurality of transaction processing apparatuses based on image information for identifying a state of each of the plurality of transaction processing apparatuses in which the customer performs the operation related to the settlement. The tracking unit tracks the movement of the customer. The associator associates the customer with the transaction processing apparatus in which the customer performs the operation relating to the settlement based on the tracked position of the customer. The notification unit notifies a payment omission in a case where a state of the transaction processing apparatus associated with the customer who moves to a predetermined specified position is not a state in which an operation related to settlement is completed.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus and an information processing system.

Background Art

[0002] Conventionally, for the registration process of purchased goods, a store clerk performs it on a registration terminal, and for the accounting process of goods, that is, the operation related to settlement, a semi-self-checkout system is known in which the customer himself / herself performs it on an accounting terminal. In recent years, a full-self-checkout system in which the customer himself / herself performs not only the accounting process but also the registration process has become widespread.

[0003] In such a self-checkout system, there is a possibility of payment omission fraud in which a customer leaves the store without completing payment after registering the goods.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the embodiments of the present invention is to provide an information processing apparatus and an information processing system that can easily notify payment omission.

Means for Solving the Problems

[0006] In one embodiment, the information processing device comprises a detection unit, a tracking unit, a linking unit, and a notification unit. The detection unit detects whether the settlement operation has been completed for each of the multiple transaction processing devices in which the customer performs settlement operations, based on image information that can identify the status of each of the multiple transaction processing devices in which the customer performs settlement operations. The tracking unit tracks the customer's movements. The linking unit links the customer with the transaction processing device in which the customer performs settlement operations, based on the location of the tracked customer. The notification unit notifies of a payment omission if the status of the transaction processing device linked to the customer who has moved to a predetermined location is not that the settlement operation has been completed. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a schematic diagram showing a self-checkout system to which an information processing system according to the first embodiment is applied, including a monitoring device to which an information processing device according to the first embodiment is applied. [Figure 2] Figure 2 is a schematic diagram showing an example of a store's checkout area. [Figure 3] Figure 3 is a perspective view showing an example of the external configuration of a self-checkout system, specifically a self-POS terminal used as a transaction processing device. [Figure 4] Figure 4 is a block diagram showing an example of the main circuit configuration of a self-service POS terminal. [Figure 5] Figure 5 is a perspective view showing an example of the external configuration of a self-checkout terminal, which is a transaction processing device included in a self-checkout system. [Figure 6] Figure 6 is a diagram illustrating the status information of a transaction processing device, represented by the color and light patterns of the warning lights provided by the device. [Figure 7] Figure 7 is a block diagram showing an example of the main circuit configuration of a monitoring device. [Figure 8] Figure 8 is a schematic diagram showing the main data structure stored in the tracking table of the monitoring device. [Figure 9]Figure 9 is a schematic diagram showing the main data structure stored in the monitoring device association table. [Figure 10] Figure 10 is a sequence diagram illustrating the fraud detection operation in a self-checkout system to which the information processing system according to the first embodiment is applied. [Figure 11] Figure 11 is a flowchart showing an example of the main steps of the tracking process performed by the monitoring device's processor. [Figure 12] Figure 12 is a flowchart showing an example of the main steps of the payment failure detection process performed by the monitoring device's processor. [Figure 13] Figure 13 shows an example of a monitoring screen displayed on the touch panel of an attendant terminal. [Figure 14] Figure 14 shows an example of a monitoring screen when an alert notification is received. [Figure 15] Figure 15 is a flowchart showing an example of the main steps of the payment omission detection process performed by the processor of a monitoring device to which the information processing device according to the second embodiment is applied. [Figure 16] Figure 16 shows an example of an idle screen displayed on the touch panel of a self-service POS terminal, which is an example of a transaction processing device in the third embodiment. [Figure 17] Figure 17 shows an example of a transaction initiation screen displayed on the touch panel of a self-service POS terminal. [Figure 18] Figure 18 shows an example of a registration screen displayed on the touch panel of a self-service POS terminal. [Figure 19] Figure 19 shows an example of a payment selection screen displayed on the touch panel of a self-service POS terminal. [Figure 20] Figure 20 shows an example of a transaction completion screen displayed on the touch panel of a self-service POS terminal. [Figure 21] Figure 21 is a schematic diagram showing a self-checkout system to which an information processing system according to the fourth embodiment is applied, including a monitoring device to which the information processing device according to the fourth embodiment is applied. [Figure 22] FIG. 22 is a schematic diagram showing an example of a checkout area of a store in the fourth embodiment.

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of an information processing apparatus and an information processing system will be described with reference to the drawings.

[0009] [First Embodiment] FIG. 1 is a schematic configuration diagram showing a self-checkout system to which an information processing system according to the first embodiment is applied. The self-checkout system includes a plurality of self-service transaction processing devices 1 and a store server 2 connected via a communication network NW such as a LAN (Local Area Network). The information processing system according to the first embodiment as an announcement system includes an attendant terminal 3, a checkout area camera 4, and a monitoring device 5 to which the information processing apparatus according to the first embodiment is applied. The monitoring device 5 is connected to the attendant terminal 3 and the checkout area camera 4. As shown by the dashed line in FIG. 1, the attendant terminal 3 may also be connected to the plurality of transaction processing devices 1 and the store server 2 via the communication network NW.

[0010] The plurality of transaction processing devices 1, the attendant terminal 3, and the checkout area camera 4 are installed in a checkout area CA of the store as shown surrounded by a one-dot chain line in FIG. 1. FIG. 2 is a schematic diagram showing an example of this checkout area CA. In the example of FIG. 2, eight transaction processing devices 1 and one attendant terminal 3 are installed in the checkout area CA. Although not limited thereto, hereinafter, the configuration and operation will be described based on the number and installation layout of the transaction processing devices 1 in this checkout area CA.

[0011] The checkout area CA has an entrance (ENT) and an exit (EXT) for shoppers. Each transaction processing device (1) at the entrance (ENT) is assigned a terminal number R1 to R8 to uniquely identify it. These terminal numbers R1 to R8 can be arbitrarily assigned by the administrator of the store server (2). A store employee (SP) is assigned to the attendant terminal (3).

[0012] The self-service transaction processing device 1 is a terminal on which customer shopping centers (SCs) perform operations related to accounting. This self-service transaction processing device 1 includes a self-service POS (Point of Sales) terminal that has product registration and payment functions, allowing the customer SC to register products and perform accounting based on the registered transaction data, and a self-service accounting terminal where the customer SC performs accounting based on product transaction data registered separately by a store employee or customer SC. The accounting terminal can be rephrased as an accounting device, payment terminal, settlement device, settlement terminal, etc. The POS terminal can be rephrased as a product registration accounting terminal, product registration accounting device, product registration payment terminal, product registration payment device, product registration settlement device, product registration settlement terminal, etc.

[0013] Figure 3 is a perspective view showing an example of the external configuration of a self-POS terminal 100 as a transaction processing device 1 included in the self-checkout system, and Figure 4 is a block diagram showing an example of the main circuit configuration of the self-POS terminal 100.

[0014] As shown in Figure 3, the self-service POS terminal 100 comprises a main unit 101 installed on the floor and a weighing unit 102 installed next to the main unit 101. The main unit 101 has a display pole 103 and a touch panel 104 attached to its upper part. The main unit 101 has a basket stand 105 in the center of the side opposite to the side where the weighing unit 102 is installed. The basket stand 105 is for customers (customer service providers) who have come from the sales floor to place their shopping baskets containing purchased items. Customer service providers stand in front of the main unit 101 in Figure 3 to perform their tasks so that they can see the screen of the touch panel 104. Therefore, from the customer service provider's perspective, the basket stand 105 is on the right side of the main unit 101, and the weighing unit 102 is on the left side. In the following description, the side where the customer service provider stands will be referred to as the front of the main unit 101, the side where the weighing unit 102 is installed will be referred to as the left side of the main unit 101, and the side where the basket stand 105 is located will be referred to as the right side of the main unit 101.

[0015] The weighing unit 102 has a structure in which a weighing pan 107 is provided on top of the housing 106, and a bag holder 108 is attached to the top of the weighing pan 107. The upper surface of the weighing pan 107 is the placement surface 109. The bag holder 108 is equipped with a pair of holding arms 110, which hold shopping bags or shopping bags brought by the customer (so-called "reusable bags"). The weighing unit 102 measures the weight of the goods placed on the placement surface 109, which are held in the shopping bags or reusable bags held by the holding arms 110. The placement surface 109 is an example of a product placement area where purchased goods to be paid for are placed. The weighing unit 102 is an example of a weighing device that measures the weight of objects placed in the product placement area.

[0016] The display pole 103 is equipped with a warning light 111 at its tip that displays status information representing the state of the self-service POS terminal 100. The warning light 111 comprises a first light-emitting unit 112 that selectively emits, for example, blue and red light, and a second light-emitting unit 113 that emits yellow light. The light emission has two types of emission patterns: continuous illumination and flashing, which alternates between illumination and extinguishing for a certain period of time. The warning light 111 can display status information by combining the color pattern and emission pattern of the first light-emitting unit 112 with the emission pattern of the second light-emitting unit 113.

[0017] The touch panel 104 consists of a display for showing various screens related to transactions such as registration and payment to the user operating the self-service POS terminal 100, and a touch sensor for detecting touch input on the screen by the user. In the self-service POS terminal 100, the user is usually a customer service center.

[0018] The front of the main unit 101 has a reading window 115 for the scanner 114 (see Figure 4), a card slot 117 for the card reader 116 (see Figure 4), and a receipt printing slot 119 for the printer 118 (see Figure 4). Furthermore, the front of the main unit 101 also has a coin slot 121, a coin outlet 122, a banknote slot 123, and a banknote outlet 124 for the automatic change dispenser 120 (see Figure 4).

[0019] Furthermore, a communication cable 125 extends from the right side of the main unit 101 to the outside, and a reader / writer 126 for electronic money media is connected to the end of this communication cable 125. The reader / writer 126 is placed on a stand 127 provided on the upper right side of the main unit 101.

[0020] As shown in Figure 4, the self-service POS terminal 100 includes, in addition to the aforementioned touch panel 104, warning light 111, scanner 114, card reader 116, printer 118, automatic change dispenser 120, and reader / writer 126, a processor 128, main memory 129, auxiliary storage device 130, communication interface 131, and system transmission line 132. The system transmission line 132 includes an address bus, data bus, control signal lines, etc. The system transmission line 132 connects the processor 128 to the other parts directly or via signal input / output circuits, and transmits data signals exchanged between them. The computer of the self-service POS terminal 100 is formed by the connection of the processor 128, main memory 129, and auxiliary storage device 130 via the system transmission line 132.

[0021] The processor 128 corresponds to the central part of the computer described above. The processor 128 controls each part to realize various functions as a self-service POS terminal 100 according to the operating system or control program. The processor 128 is, for example, a CPU (Central Processing Unit). The processor 128 may also be, for example, an MPU (Micro Processing Unit), SoC (System on a Chip), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array). Alternatively, the processor 128 may be a combination of several of these.

[0022] Main memory 129 corresponds to the main memory portion of the computer described above. Main memory 129 includes a non-volatile memory area and a volatile memory area. In the non-volatile memory area, main memory 129 stores the operating system or control programs. Main memory 129 may also store data necessary for the processor 128 to perform processing to control each part in the non-volatile or volatile memory area. Main memory 129 uses the volatile memory area as a work area where data is rewritten as needed by the processor 128. The non-volatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory). For example, main memory 129 stores the transaction file 133.

[0023] Transaction file 133 is a data file for storing transaction data related to a single commercial transaction processed by the self-service POS terminal 100. Transaction file 133 stores data such as transaction number, purchased item data, total number of items, total amount, discount amount, and settlement amount. The transaction number is a series of numbers issued each time a commercial transaction is processed by the self-service POS terminal 100. Purchased item data is record data created for each item bought and sold in the commercial transaction identified by the transaction number. Here, purchased item data consists of items such as product code, product name, price, number of items, and amount. The number of items is the number of items purchased for the product identified by the product code. The amount is the amount for the number of items purchased. Transaction file 133 can store multiple purchased item data. The total number of items is the sum of the number of items in each purchased item data. The total amount is the sum of the amounts in each purchased item data. The discount amount is the discount amount applied to the total amount. The settlement amount is the total amount minus the discount amount.

[0024] The auxiliary storage device 130 corresponds to the auxiliary storage portion of the computer described above. For example, EEPROM (Electric Erasable Programmable Read-Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive) can be the auxiliary storage device 130. The auxiliary storage device 130 stores data used by the processor 128 in performing various processes, data created by the processing performed by the processor 128, etc. In addition, the auxiliary storage device 130 may also store the control program described above.

[0025] The communication interface 131 communicates data with external devices connected via the communication network NW according to a pre-configured communication protocol. The external devices are, for example, the store server 2, and may also include the attendant terminal 3.

[0026] The scanner 114 reads a code symbol from the product held over the reading window 115. Each product sold in the store is marked with a code symbol that encodes a product ID or other information to identify the product. The code symbol is, for example, a barcode. The code symbol may also be, for example, a two-dimensional data code. The scanner 114 may be a type that reads the code symbol by scanning with a laser beam, or a type that reads the code symbol from an image captured by an imaging device.

[0027] The card reader 116 reads card data recorded on card media such as credit cards and point cards. The card reader 116 pulls the card media inserted into the card slot 117 into the main unit 101, reads the card data, and then ejects it from the card slot 117.

[0028] The printer 118 prints receipt data, etc., representing the details of a commercial transaction, onto receipt paper. The receipt paper with the printed receipt data is ejected from the output slot 119, cut by a cutter (not shown), and issued as a receipt or invoice.

[0029] The automatic change dispenser 120 includes a coin unit 134 and a banknote unit 135. The coin unit 134 sorts the coins inserted into the coin slot 121 one by one, identifies the denomination, and stores them in a safe according to their denomination. The coin unit 134 retrieves the coins of the corresponding denomination from the safe, for example based on change data, and dispenses them to the coin outlet 122. The banknote unit 135 sorts the banknotes inserted into the banknote slot 123 one by one, identifies the denomination, and stores them in a safe according to their denomination. The banknote unit 135 retrieves the banknotes of the corresponding denomination from the safe, for example based on change data, and dispenses them to the banknote outlet 124.

[0030] The reader / writer 126 reads and rewrites electronic money recorded on the electronic money medium. The electronic money medium is, for example, a contactless IC card. The electronic money medium may also be an electronic device such as a smartphone or tablet.

[0031] The processor 128 implements the processing unit for well-known product registration and payment. Each part implemented in the processor 128 can also be called a function. Each part implemented in the processor 128 can also be said to be implemented in the control unit, which includes the processor 128 and the main memory 129.

[0032] Thus, the self-service POS terminal 100 has a settlement function that allows payment using either cash or cashless methods. Cashless payment methods include credit card payments, electronic money payments, point payments, and code payments. Customers who have purchased items in a shopping basket or cart in the sales area and moved to the checkout area (CA) operate the self-service POS terminal 100 to register the purchased items and perform accounting processing for the transaction data of the registered purchased items. In this way, customers can complete the settlement process for transactions from item registration to payment themselves.

[0033] Figure 5 is a perspective view showing an example of the external configuration of a self-checkout terminal 150, which is a transaction processing device 1 included in the self-checkout system. Configurations similar to those of the self-POS terminal 100 are given the same reference numerals, and their explanations are omitted.

[0034] The self-checkout terminal 150 comprises a first enclosure 151 and a second enclosure 152.

[0035] The front of the first enclosure 151 is arranged the coin slot 121, coin outlet 122, banknote slot 123, and banknote outlet 124 of the automatic change dispenser 120, as well as the scanner reading window 115 of the scanner 114. Note that the scanner 114 is not used to read code symbols from products like the self-service POS terminal 100, but is used to read code symbols for code payments.

[0036] The second enclosure 152 is mounted on the top surface 153 of the first enclosure 151 and is equipped with a printer 118. The printer 118 has a receipt printing slot 119 located on the front surface 154 of the second enclosure 152.

[0037] A card reader 116 with a card slot 117 is located to the left of the second enclosure 152, on the top surface 153 of the first enclosure 151. A touch panel 104 is mounted above the second enclosure 152.

[0038] Furthermore, the self-checkout terminal 150 is equipped with a cylindrical display pole 103 erected on the rear side of the upper surface 153 of the first housing 151. The display pole 103 has a warning light 111 at its upper tip, which includes a first light-emitting unit 112 and a second light-emitting unit 113. The display pole 103 further has a display unit 155 below the warning light 111 that displays images and characters. The display unit 155 is made up of, for example, a liquid crystal panel. The display unit 155 displays, for example, the details of an error that occurred in the self-checkout terminal 150.

[0039] For example, on the right side of the self-checkout terminal 150 (when facing it), there is a basket stand 105 for placing shopping baskets, etc.

[0040] Attendant terminal 3 is a terminal operated by an attendant employee SP who is a supervisor monitoring the checkout area CA. Attendant terminal 3 may be a personal computer connected to monitoring device 5 by wire or wireless connection, or a tablet terminal that can be connected to monitoring device 5 wirelessly and is portable by the employee.

[0041] Furthermore, the attendant terminal 3 may have the function of connecting to a communication network NW and monitoring multiple transaction processing devices 1. In addition, the attendant terminal 3 may have the same configuration as the self-service POS terminal 100. In the attendant terminal 3 with monitoring functions, the status of each self-service POS terminal 100 is stored in the main memory 129. For example, the attendant terminal 3 can store in the main memory 129 a list of purchased items registered in each self-service POS terminal 100, error status, etc. The attendant terminal 3 displays the status of each self-service POS terminal 100 stored in the main memory 129 on the display of the touch panel 104. In addition to cash and cashless payment functions, the attendant terminal 3 may also have functions to handle payments that require human intervention, such as payments using gift certificates or vouchers that cannot be machine-readable, returns, and refunds.

[0042] The store server 2 is equipped with a product master database and a sales data database, and provides various services related to the self-checkout system to each self-POS terminal 100 connected via a communication network NW. The product master database stores product data for each product sold in the store, with one record for each product, such as product code, product name, unit price, and classification name. The product code is a unique code set for each product to identify it individually. The sales data database stores sales data generated in each transaction, which is transmitted from each transaction processing unit 1 when payment is completed, i.e., when the transaction ends. The sales data database stores transaction data representing the transaction details of the products subject to payment and payment data related to the settlement of that transaction, with one record for each transaction. The transaction data includes the product name (product code) and price of the products subject to payment, and the total amount of each product. The payment data includes the amount paid by the customer and the change amount if payment is made in cash, and information on the payment method if payment is made cashless. Furthermore, sales data may include elements such as member code, company code (company name) indicating the company operating the store, store code (store name), store phone number, address, transaction date and time, transaction number (receipt number), terminal number, and name of the person in charge. Note that the elements included in sales data are not limited to these examples and may include other elements, or elements such as phone numbers and addresses may be removed.

[0043] In the transaction processing device 1, for example, if it is a self-service POS terminal 100, the scanner 114 reads a code symbol printed or attached to the product to obtain the product code, reads product data from the product data file held by the transaction processing device 1 for the product identified by that product code, and registers the product data as transaction data, thereby updating and storing sales data including transaction data and settlement data. The product data file may be a copy of the product master database held by the store server 2. In the case where the transaction processing device 1 is a self-service checkout terminal 150, it receives and stores transaction data registered by a registration terminal (not shown). Alternatively, the scanner of the self-service checkout terminal 150 reads a paper medium on which a code indicating transaction data registered by a registration terminal (not shown) is printed. As a result, the self-service checkout terminal 150 registers the transaction data and proceeds to the subsequent accounting process. In this case, instead of a paper medium on which a code indicating transaction data registered by a registration terminal (not shown) is printed, the code indicating transaction data may be displayed on the screen of a mobile terminal held by the consumer, and the transaction data may be registered by having the scanner of the self-service checkout terminal 150 read this code, and proceeding to the subsequent accounting process. Once accounting is complete, that is, when a transaction is finished, the transaction processing unit 1 prints the sales data onto paper to issue a receipt for that transaction, and also sends the sales data for that transaction to the store server 2. The sales data database on the store server 2 stores the sales data that has been sent from each transaction processing unit 1 in this manner.

[0044] As shown in Figure 2, the checkout area camera 4 is positioned on the ceiling of the checkout area CA, etc., to photograph the entire checkout area CA and transmit the captured camera image to the monitoring device 5. The checkout area camera 4 is, for example, a panoramic camera or fisheye camera capable of 360-degree shooting, and is installed so that all of the multiple transaction processing devices 1 located in the checkout area CA, specifically all of the warning lights 111 of the self-service POS terminals 100, are within its imaging range. If all the warning lights 111 cannot be photographed by one checkout area camera 4, there may be multiple checkout area cameras 4. The checkout area camera 4 is an example of an acquisition device that acquires image information capable of identifying the status of each of the multiple transaction processing devices 1. The checkout area camera 4 may utilize an existing monitoring camera installed in the checkout area CA.

[0045] The monitoring device 5 detects the status of each transaction processing device 1 based on the image of the warning light 111 of each transaction processing device 1 included in the camera image captured by the checkout area camera 4. Figure 6 is a diagram illustrating the status information of the transaction processing devices 1, i.e., the self-POS terminals 100 and self-checkout terminals 150, as represented by the color pattern and light emission pattern of the warning light 111. In Figure 6, solid hatching indicates that the device is lit, and dashed hatching indicates that it is blinking. The warning light 111 indicates that it is in operation by blinking the first light-emitting part 112 in blue, and indicates that it is idle by illuminating it in blue. Therefore, the monitoring device 5 can determine the status of each self-POS terminal by recognizing the color pattern and light emission pattern of the warning light 111 of each self-POS terminal 100 from the camera image captured by the checkout area camera 4. In this way, the warning light 111 can indicate whether the transaction processing device 1 is idle or in operation by at least one of the light emission color and light emission state. Here, the "operating" state means that the customer SC has not yet made a payment, and the "idle" state means that the previous customer SC has made a payment and is waiting to be used by the next customer SC, that is, the previous customer SC has paid. Therefore, the warning light 111 is an example of a display member that indicates whether the customer SC using or having used the transaction processing device 1 is in an unpaid or paid state. In this embodiment, the warning light 111 is given as an example having two light-emitting units, a first light-emitting unit 112 and a second light-emitting unit 113, but it may also be a single light-emitting unit that indicates whether the state is unpaid or paid.

[0046] Figure 7 is a block diagram showing an example of the main circuit configuration of the monitoring device 5. As shown in Figure 7, the monitoring device 5 comprises a processor 51, main memory 52, auxiliary storage device 53, communication interface 54, input / output device 55, and system transmission path 56. The system transmission path 56 includes an address bus, data bus, control signal lines, etc. The system transmission path 56 connects the processor 51 to the other parts directly or via signal input / output circuits and transmits data signals exchanged between them. The computer of the monitoring device 5 is formed by the connection of the processor 51, main memory 52, and auxiliary storage device 53 via the system transmission path 56. The computer is an example of an information processing device according to the first embodiment.

[0047] The processor 51 corresponds to the central part of the computer described above. The processor 51 controls each part to realize various functions as a monitoring device 5 according to the operating system or application program. The processor 51 is, for example, a CPU. The processor 51 may be multi-core / multi-threaded and capable of executing multiple processes in parallel. The processor 51 may be, for example, an MPU, SoC, DSP, GPU, ASIC, PLD, or FPGA. Alternatively, the processor 51 may be a combination of several of these.

[0048] Main memory 52 corresponds to the main memory portion of the computer described above. Main memory 52 includes a non-volatile memory area and a volatile memory area. In the non-volatile memory area of ​​main memory 52, the operating system or application programs are stored. The application programs include, for example, information processing programs that cause the computer to function as an information processing device according to the first embodiment. Main memory 52 may also store data necessary for the processor 51 to perform processing to control each part in non-volatile or volatile memory areas. Main memory 52 uses the volatile memory area as a work area where data is rewritten as appropriate by the processor 51. The non-volatile memory area is, for example, ROM. The volatile memory area is, for example, RAM. Auxiliary storage device 53 corresponds to the auxiliary storage portion of the computer described above. For example, EEPROM, HDD, or SSD can be auxiliary storage device 53. Auxiliary storage device 53 stores data used by the processor 51 in performing various processes, data created by processing by the processor 51, etc. Auxiliary storage device 53 may also store the control programs described above.

[0049] The communication interface 54 may be a general-purpose communication device that communicates data with the attendant terminal 3 via wired or wireless connection. The communication interface 54 may also include a general video interface function for inputting camera footage from the checkout area camera 4.

[0050] The input / output device 55 is a user interface that includes input devices such as a keyboard and mouse, and display devices such as an LCD display. The monitoring device 5 does not necessarily have to have an input / output device 55. In this case, an information processing terminal (not shown) that can communicate via the communication interface 54 can be used as the user interface. The information processing terminal may include, for example, a personal computer, smartphone, or tablet computer. Furthermore, by providing the communication interface 54 with the function to communicate with the store server 2 via the communication network NW, the user interface provided by the store server 2 can be used.

[0051] The monitoring device 5 includes, for example, a setting storage unit 531, a camera image temporary storage unit 532, a tracking table 533, and a linking table 534 in its auxiliary storage device 53.

[0052] The setting storage unit 531 stores the terminal number assigned to each transaction processing device 1. Furthermore, the setting storage unit 531 stores a predetermined location for detecting a customer SC who attempts to leave the store without settling their bill. This predetermined location can be set in advance by the administrator of the monitoring device 5, for example, using an input / output device 55. For example, the predetermined location can be set to be outside the rectangular predetermined range RR, as shown enclosed by a dashed line in Figure 2, which is set in advance with respect to the checkout area CA. The shape of the predetermined range RR can be arbitrarily set according to the arrangement of the checkout area CA. In addition, the predetermined location may be set to be on the entrance ENT or exit EXT side with respect to a predetermined line arbitrarily set in the checkout area CA. That is, the predetermined location can be a position beyond the predetermined line on the entrance ENT or exit EXT side. The predetermined line may be, for example, one side of the predetermined range RR on the entrance ENT or exit EXT side. Furthermore, the setting storage unit 531 stores installation location information that is uniquely determined based on the arrangement of each transaction processing device 1 in the checkout area CA, such as the position of the warning light 111 of each transaction processing device 1 and the position of the operating range where the customer SC operating each transaction processing device 1 should be located. In addition, the setting storage unit 531 can also store installation location information for the attendant terminal 3.

[0053] The camera image temporary storage unit 532 temporarily stores camera images captured by the checkout area camera 4 for a certain period of time.

[0054] The tracking table 533 stores the location of each customer SC who enters the checkout area CA. Figure 8 is a schematic diagram showing the main data structure stored in this tracking table 533. Note that the data structure shown in Figure 8 is just one example. The format is not particularly limited as long as the processor 51 can obtain the necessary information. As shown by the dashed line in Figure 2, the processor 51 sets a rectangular ROI (Region of Interest) which is a region of interest from which to extract images of each customer SC that enters the checkout area CA with respect to the camera image captured by the checkout area camera 4, and assigns an ROI number to distinguish each region of interest. Therefore, the ROI number can also be said to be the customer number. As shown in Figure 8, for example, the tracking table 533 stores the coordinates in the camera image captured by the checkout area camera 4 as the location of the customer SC for each ROI number of each customer SC that enters the checkout area CA. The coordinates are not limited to this, but in the example in Figure 8, they are the coordinates of the two diagonal points of the rectangular ROI.

[0055] The linking table 534 stores the correspondence between customer SCs and transaction processing devices 1. Figure 9 is a schematic diagram showing the main data structure stored in this linking table 534. Note that the data structure shown in Figure 9 is just one example. The format is not particularly limited as long as the information necessary for the processor 51 can be obtained. For example, as shown in Figure 9, the linking table 534 stores the terminal number assigned to the transaction processing device 1 for each ROI number that identifies the customer SC. Note that the same terminal number may be stored for multiple ROI numbers. This is a situation such as when a married couple is shopping together.

[0056] Next, we will describe the various components implemented in the processor 51 of the monitoring device 5. As shown in Figure 7, the processor 51 implements, for example, a tracking unit 511, a linking unit 512, a terminal status detection unit 513, and a fraud detection unit 514. Each component implemented in the processor 51 can also be called a function. Each component implemented in the processor 51 can also be said to be implemented in the control unit, which includes the processor 51 and the main memory 52.

[0057] The tracking unit 511 receives camera images from the checkout area camera 4 via the communication interface 54 and stores the received camera images in the camera image temporary storage unit 532. The camera images include images of customers moving within the checkout area CA, as well as images of the warning lights 111 on each transaction processing device 1. The tracking unit 511 stores camera images in the camera image temporary storage unit 532 for a period of time that allows it to determine whether the warning lights 111 are flashing or lit. In this way, the tracking unit 511 can acquire image information that allows it to identify the status of each transaction processing device 1 on which a customer SC performs settlement operations. Furthermore, the tracking unit 511 tracks the movement of each customer SC in the checkout area CA from the camera images stored in the camera image temporary storage unit 532 for a certain period of time and stores the current location information of each customer SC in the tracking table 533. Therefore, the tracking unit 511 is an example of a tracking unit that tracks the movement of customer SCs.

[0058] The linking unit 512 identifies customer SCs within the operating range of each transaction processing unit 1 based on the location information of each customer SC stored in the tracking table 533 and the installation location information of each transaction processing unit 1 stored in the setting storage unit 531. The linking unit 512 then registers the identified customer SCs that operate each transaction processing unit 1 in the linking table 534. The linking unit 512 is an example of a linking unit that links customer SCs with transaction processing units 1 in which customer SCs perform settlement operations, based on the location of the customer SCs being tracked.

[0059] The terminal status detection unit 513 detects the status of each transaction processing unit 1 based on camera images stored in the camera image temporary storage unit 532 for a certain period of time. Specifically, the camera images include images of the warning lights 111 of each transaction processing unit 1. In other words, the terminal status detection unit 513 acquires image information from the camera image temporary storage unit 532 that allows for the identification of the status of each transaction processing unit 1 in which a customer SC performs settlement operations. The terminal status detection unit 513 determines the status information of each transaction processing unit 1, represented by the color pattern and emission pattern of the warning light 111, from the images of the warning lights 111 of each transaction processing unit 1 captured in the acquired camera images for a certain period of time. Based on the determined status information, the terminal status detection unit 513 determines whether the transaction processing unit 1 is in use or idle. Idle status, as mentioned above, indicates a state in which the previous customer SC has paid, that is, a state in which settlement operations have been completed. Thus, the terminal status detection unit 513 is an example of a detection unit that detects whether or not the settlement operation has been completed for each of the multiple transaction processing units 1 in which the customer SC performs settlement operations, based on image information that can identify the status of each of the multiple transaction processing units 1.

[0060] The fraud detection unit 514 determines whether or not a payment failure has occurred based on the location information of each customer SC stored in the tracking table 533, the linking information between each customer SC and the transaction processing unit 1 stored in the linking table 534, and the status of each transaction processing unit 1 detected by the terminal status detection unit 513. Specifically, the fraud detection unit 514 determines that no fraud has occurred if the status of the transaction processing unit 1 linked to a customer SC that is in a specified location, i.e., a customer SC that has gone outside the specified range RR or crossed the specified line on the entrance gate ENT or exit gate EXT side, is idle, i.e., the settlement operation has been completed. Conversely, the fraud detection unit 514 determines that a payment failure has occurred if the status of the transaction processing unit 1 linked to a customer SC that is in a specified location is in operation, i.e., the settlement operation has not been completed. When it determines that a payment failure has occurred, the fraud detection unit 514 sends a payment failure notification to the attendant terminal 3 via the communication interface 54. The fraud detection unit 514 is an example of a notification unit that notifies of a payment omission if the status of the transaction processing unit 1 associated with a customer SC that has moved to a predetermined location is not in a state where settlement operations have been completed. A payment omission can be rephrased as accounting omission, unaccounted for, settlement omission, unsettled, settlement omission, unsettled, etc.

[0061] The monitoring device 5 can be implemented, for example, by using a general-purpose computer as hardware and writing the control program to the main memory 52 or auxiliary storage device 53. The control program may be stored in the main memory 52 or auxiliary storage device 53 when the monitoring device 5 is transferred, or it may be transferred separately from the general-purpose computer mentioned above. In the latter case, the control program may be recorded on a removable recording medium such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory, or transferred via a network.

[0062] The operation of the monitoring device 5 and the information processing system with the above configuration will be described below.

[0063] Figure 10 is a sequence diagram illustrating the fraud detection operation in a self-checkout system to which the information processing system according to the first embodiment is applied. Note that Figure 10 shows only one of the multiple transaction processing devices 1 as a representative example.

[0064] As shown in Figure 10, the monitoring device 5 acquires camera images from the checkout area camera 4 (step S1).

[0065] Then, the monitoring device 5 tracks each customer SC within the checkout area CA using the camera footage (step S2). The monitoring device 5 updates the current location of each customer SC stored in the tracking table 533 based on the current location of each customer SC that it has tracked. In addition, if an untracked customer SC, i.e., a new customer SC enters the checkout area CA, the monitoring device 5 assigns a new ROI number to it and registers its current location in the tracking table 533.

[0066] Furthermore, the monitoring device 5 updates the link between each customer SC and the transaction processing device 1 based on the positional relationship between each customer SC and each transaction processing device 1 (step S3). That is, if a customer SC that has not been linked enters the operating range of any transaction processing device 1, the monitoring device 5 registers the link between that customer SC and the transaction processing device 1 in the link tracking table 533.

[0067] Furthermore, the monitoring device 5 detects the status of each transaction processing device 1 based on the image of the warning light 111 of each transaction processing device 1 (step S4).

[0068] Then, the monitoring device 5 determines whether or not there is any fraudulent payment (step S5). Specifically, for each transaction processing device 1 in which the detected situation is in operation, that is, not in a state where settlement operations have been completed, the monitoring device 5 checks the customer SC associated with that transaction processing device 1 using the linked tracking table 533 and obtains the current location of that customer SC from the tracking table 533. Then, it determines whether or not there is fraud based on whether or not the obtained current location of the customer SC is in a specified location, that is, whether or not it is outside the specified range RR, or whether or not it has crossed the specified line. For example, if transaction processing devices 1 with terminal numbers R2, R3, R7, and R8 are in operation, the location of the customer SC associated with those transaction processing devices 1 is checked. In this case, if the situation is as shown in Figure 2, the customer SC associated with those transaction processing devices 1 is not in a specified location, that is, it is within the specified range RR, or has not crossed the specified line, so it is determined that no fraud has occurred.

[0069] The operations in steps S1 to S5 are repeated thereafter. In Figure 10, steps S4 and S5 are performed immediately after steps S1 to S3, but steps S4 and S5 may be performed after each multiple execution of steps S1 to S3. In other words, just because a customer SC leaves the operating range of each transaction processing device 1 does not mean that the customer SC has immediately gone outside the specified range RR or crossed the specified line. Therefore, there is no need to have a one-to-one relationship between the operations in steps S1 to S3 and the operations in steps S4 and S5.

[0070] Furthermore, for example, if, in addition to the transaction processing devices 1 of terminal numbers R2, R3, R7, and R8 mentioned above, the transaction processing device 1 of terminal number R1 is also in operation, then in the situation shown in Figure 2, the customer SC associated with the transaction processing device 1 of terminal number R1 is in the specified position, that is, outside the specified range RR, or has crossed the specified line, so the monitoring device 5 determines in step S5 that a payment failure has occurred. In such a case, the monitoring device 5 sends an alert notification to the attendant terminal 3 (step S6). The alert notification includes information indicating the transaction processing device 1 where the payment failure occurred, in this example, terminal number R1. The alert notification may also include an image of the customer SC in the specified position, extracted from the camera footage.

[0071] When the attendant terminal 3 receives an alert notification from the monitoring device 5, it displays the alert notification, for example, as a pop-up (step S7). After the employee SP confirms this alert notification, they perform a predetermined stop operation on the attendant terminal 3 to stop the alert notification (step S8), and then take necessary actions such as alerting the relevant customer SC. In response to this predetermined stop operation, the attendant terminal 3 sends a terminal status update instruction to the relevant transaction processing device 1 (step S9).

[0072] Upon receiving this terminal status update instruction, the transaction processing device 1 updates the status to idle and changes the indicator light 111 to show idle (step S10).

[0073] The operation of the monitoring device 5 to achieve this operation will be described below. Figure 11 is a flowchart showing an example of the main steps of the tracking process performed by the processor 51 of the monitoring device 5. In this specification, unless otherwise specified, the processing operation of the processor 51 is assumed to transition from ACTn (where n is a natural number) to ACT(n+1). Also, the procedure shown in Figure 10 is just one example. The procedure is not particularly limited as long as similar results can be obtained.

[0074] As ACT101, the tracking unit 511 implemented in the processor 51 acquires camera images transmitted from the checkout area camera 4 via the communication network NW through the communication interface 54. The tracking unit 511 stores the acquired camera images in the camera image temporary storage unit 532.

[0075] As ACT102, the tracking unit 511 uses known motion detection and human detection technologies to detect people within the checkout area CA from temporarily stored camera footage.

[0076] As ACT103, the tracking unit 511 determines whether or not a new person has been detected. Specifically, the tracking unit 511 compares the location information of each detected person with the location information stored in the tracking table 533 and determines whether or not there is a person whose location information is not within a distance, i.e., a number of pixels, determined based on the detection time interval. The range within the checkout area CA where this determination is made may also be limited to the vicinity of the entrance ENT. If there is no new detection, the tracking unit 511 determines NO in ACT103 and proceeds to the process of ACT105. If there is a new detection, the tracking unit 511 determines YES in ACT103 and proceeds to the process of ACT104.

[0077] As ACT104, the tracking unit 511 adds the newly detected person to the tracking target. Specifically, the tracking unit 511 assigns an ROI number not stored in the tracking table 533 to the newly detected person, who is a new entering customer SC, and registers that ROI number in the tracking table 533.

[0078] As ACT105, the tracking unit 511 updates the location information of each customer SC stored in the tracking table 533 based on the current location of each person it has detected.

[0079] As ACT106, the linking unit 512 implemented in the processor 51 determines whether there are any new intruders within the operating range of each transaction processing unit 1. Specifically, for each transaction processing unit 1 whose terminal number is not stored in the linking table 534, the linking unit 512 determines, based on the location information stored in the tracking table 533, whether there are any customer SCs that have entered the operating range stored in the setting storage unit 531. If there are no new intruders, the linking unit 512 determines NO in ACT106 and proceeds to the process of ACT101. If there are new intruders, the linking unit 512 determines YES in ACT106 and proceeds to the process of ACT107.

[0080] As ACT107, the linking unit 512 registers a new link. Specifically, the processor 51 stores in the linking table 534 the terminal number of the transaction processing unit 1 that is determined to have a new intruder, corresponding to the ROI number of the customer SC that is determined to be the new intruder. After that, the linking unit 512 proceeds to the processing of ACT101.

[0081] Figure 12 is a flowchart showing an example of the main steps of the payment failure detection process performed by the processor 51 of the monitoring device 5. The processor 51 performs this process at regular intervals, for example, every few seconds, based on a control program stored in the main memory 52 or auxiliary storage device 53. The procedure shown in Figure 12 is an example; the procedure is not particularly limited as long as similar results can be obtained.

[0082] As ACT201, the terminal status detection unit 513 implemented in the processor 51 initializes the value of an internal counter n (not shown) configured in the processor 51 to "1". This value of internal counter n is used to specify the terminal number of the transaction processing unit 1.

[0083] As ACT202, the terminal status detection unit 513 acquires an image of the warning light 111 of the transaction processing device 1 with terminal number n from the camera images stored in the camera image temporary storage unit 532 for a certain period of time, and detects the terminal status of the transaction processing device 1 based on the status information of the transaction processing device 1 represented by the color pattern and light emission pattern of the warning light 111.

[0084] As ACT203, the terminal status detection unit 513 determines whether the detected terminal status is in operation or not. If it is not in operation, i.e., idle, the terminal status detection unit 513 determines NO in ACT203 and proceeds to the process of ACT209. If it is in operation, the terminal status detection unit 513 determines YES in ACT203 and proceeds to the process of ACT204.

[0085] As ACT204, the fraud detection unit 514 implemented in the processor 51 checks the ROI number, i.e., the customer number, associated with the transaction processing unit 1 of terminal number n using the association table 534.

[0086] As ACT205, the fraud detection unit 514 confirms the location of the customer SC corresponding to the confirmed ROI number. Specifically, the fraud detection unit 514 reads the location information corresponding to the confirmed ROI number from the tracking table 533.

[0087] As ACT206, the fraud detection unit 514 determines whether the customer SC has moved to a predetermined location stored in the setting storage unit 531. Specifically, the fraud detection unit 514 determines whether the current location of the customer SC, indicated by the confirmed location information, is outside the predetermined range RR stored in the setting storage unit 531. Alternatively, it may determine whether the current location of the customer SC, indicated by the location information, has crossed a predetermined line. If the customer SC has moved to a predetermined location, the fraud detection unit 514 determines YES in ACT206 and proceeds to the process of ACT215. If the customer SC has not moved to a predetermined location, the fraud detection unit 514 determines NO in ACT206 and proceeds to the process of ACT207.

[0088] In ACT207, the fraud detection unit 514 determines whether the value of the internal counter n is "8", that is, whether it has detected a situation for all transaction processing devices 1 provided by the self-checkout system. If the value of the internal counter n is "8", the fraud detection unit 514 determines YES in ACT207 and terminates this process. If the value of the internal counter n is not "8", the fraud detection unit 514 determines NO in ACT207 and proceeds to the process of ACT208.

[0089] As ACT208, the terminal status detection unit 513 increments the value of the internal counter n by "+1". After that, the terminal status detection unit 513 proceeds to the process of ACT202.

[0090] As ACT209, the terminal status detection unit 513 determines whether there is a link in the linking table 534 for the transaction processing device 1 of terminal number n, which has been determined to be idle. Specifically, the terminal status detection unit 513 determines whether terminal number n is stored in the linking table 534. If there is no link, the terminal status detection unit 513 determines NO in ACT209 and proceeds to the process of ACT207. If there is a link, the terminal status detection unit 513 determines YES in ACT209 and proceeds to the process of ACT210.

[0091] As ACT210, the terminal status detection unit 513 confirms the ROI number, i.e., the customer number, associated with the transaction processing device 1 of terminal number n using the association table 534.

[0092] As ACT211, the terminal status detection unit 513 confirms the location of the customer SC corresponding to the confirmed ROI number. Specifically, the terminal status detection unit 513 reads the location information corresponding to the confirmed ROI number from the tracking table 533.

[0093] As ACT212, the terminal status detection unit 513 determines whether the customer SC has moved to a predetermined location stored in the setting storage unit 531. Specifically, the terminal status detection unit 513 determines whether the current location of the customer SC, indicated by the confirmed location information, is outside the predetermined range RR stored in the setting storage unit 531. It may also determine whether the customer SC's position has crossed a predetermined line. If the customer SC has not moved to a predetermined location, the terminal status detection unit 513 determines NO in ACT212 and proceeds to the process of ACT207. If the customer SC has moved to a predetermined location, the terminal status detection unit 513 determines YES in ACT212 and proceeds to the process of ACT213.

[0094] As ACT213, the terminal status detection unit 513 deletes the confirmed ROI number and its corresponding location information from the tracking table 533.

[0095] As ACT214, the terminal status detection unit 513 deletes the terminal number n stored in the association table 534. After that, the terminal status detection unit 513 proceeds to the process of ACT207.

[0096] For example, suppose that the transaction processing unit 1 for terminal number R1 detected in the ACT203 process is idle, and terminal number R1 is associated with customer SC of ROI number ROI1, as shown in Figure 9. This corresponds to the situation shown in Figure 2, where customer SC of ROI number ROI1 has completed the settlement operations at the transaction processing unit 1 for terminal number R1 and is heading towards the exit EXT of the checkout area CA. In such a case, the process proceeds from ACT203 to ACT209, and from ACT209 through ACT210 and ACT211, the customer SC is confirmed to be in the specified location in the ACT212 process. Then, in the ACT213 and ACT214 processes, the record for ROI number ROI1 that stores the location information of the customer SC in the tracking table 533 is deleted, and the association between ROI number ROI1 and terminal number R1, i.e., the corresponding record, is deleted from the association table 534.

[0097] As ACT215, the fraud detection unit 514 implemented in the processor 51 sends an alert notification to the attendant terminal 3 via the communication interface 54. Subsequently, the fraud detection unit 514 proceeds to the processing of ACT207.

[0098] For example, suppose that the transaction processing unit 1 for terminal number R1 detected in ACT203 is in operation, and terminal number R1 is associated with customer SC of ROI number ROI1, as shown in Figure 9. This corresponds to a situation where, as shown in Figure 2, customer SC of ROI number ROI1 has left the defined range RR and is heading towards the exit EXT of the checkout area CA, but the settlement operation at transaction processing unit 1 for terminal number R1 has not yet been completed, meaning there is a payment omission. In such a case, the process proceeds from ACT203 through ACT204 and ACT205 to ACT206, and once it is confirmed that the customer SC has left the defined range RR, an alert notification is sent in ACT215.

[0099] Figure 13 shows an example of a monitoring screen 31 displayed on the touch panel of an attendant terminal 3 that has the function of being connected to a communication network NW and monitoring multiple transaction processing devices 1. The monitoring screen 31 includes a terminal list area 32 that displays a list of multiple transaction processing devices 1 to be monitored as icons, and a terminal status area 33 that displays the display contents of the touch panel 104 of each transaction processing device 1. The attendant terminal 3 receives status information indicating the status of each transaction processing device 1 transmitted from each transaction processing device 1, and display information displayed on the touch panel 104 of the transaction processing device 1. Based on the received status information, the attendant terminal 3 displays the current status of each transaction processing device 1 in an identifiable manner by changing the color, density, or shape of the icons displayed in the terminal list area 32. In the example in Figure 13, transaction processing devices 1 with terminal numbers R1, R2, R3, R7, and R8 are displayed with icons indicating that they are in operation, and transaction processing devices 1 with terminal numbers R4, R5, and R6 are displayed with icons indicating that they are idle. Furthermore, the attendant terminal 3 displays a reduced image of the screen of each touch panel 104 in the terminal status area 33, based on the display information shown on the touch panel 104 of each transaction processing device 1 that it has received.

[0100] Figure 14 shows an example of the monitoring screen 31 when an alert notification is received from the monitoring device 5. As mentioned above, the alert notification includes information indicating the transaction processing device 1 where the payment failure occurred, and may include an image of the corresponding customer SC extracted from the camera footage. Upon receiving such an alert notification, the attendant terminal 3 superimposes a pop-up alert 34 based on the alert notification onto the display content of each transaction processing device 1 shown in the terminal status area 33 of the monitoring screen 31, as shown in Figure 14. The pop-up alert 34 may include a customer image 35 along with a message indicating which transaction processing device 1 the payment failure occurred in. Furthermore, the pop-up alert 34 includes an "OK" button 36 for the attendant employee SP to input that they have confirmed the occurrence of this payment failure. It goes without saying that the pop-up alert 34 shown in Figure 14 is just one example. In addition, in response to receiving the alert notification, the attendant terminal 3 changes the icon of the corresponding terminal number displayed in the terminal list area 32, as shown in Figure 14, to an icon indicating that a payment failure has occurred.

[0101] When the "OK" button 36 in the pop-up alert 34 is pressed, the attendant terminal 3 sends a terminal status update instruction to the transaction processing unit 1, which has been determined to have a payment failure based on the alert notification. In other words, pressing the "OK" button 36 is an example of a predetermined stop operation that instructs the system to stop the alert notification. As mentioned above, the transaction processing unit 1 that receives this terminal status update instruction, for example, the transaction processing unit 1 of terminal number R1 in the above example, updates its status to idle and changes the warning light 111 to indicate idle.

[0102] If the attendant terminal 3 is a general personal computer or tablet terminal that does not have the function of monitoring the transaction processing device 1, the same display as the pop-up alert 34 will be shown on its screen. In this case, the "OK" button 36 will be a button that instructs the display of the pop-up alert 34 to end. The actual status update of the transaction processing device 1 and the change of the display of the warning light 111 will be performed by the attendant employee SP going to the location where the transaction processing device 1 is installed and performing the prescribed update operation.

[0103] Of course, the attendant terminal 3 does not have the function to monitor the transaction processing device 1, but it may be given the function to connect to the communication network NW and update the status of the transaction processing device 1.

[0104] By updating the status of the transaction processing device 1 in this way, the monitoring device 5, in the next payment failure detection process executed after a certain period of time, will find that the status of the transaction processing device 1 for terminal number R1 detected in ACT202 is idle, and will proceed from the process of ACT203 to the process of ACT209. Then, in the process of ACT212, via the processes of ACT209, ACT210 and ACT211, it is confirmed that the customer SC is in the specified location. Then, in the processes of ACT213 and ACT214, the record of ROI number ROI1 that stores the location information of the customer SC in the tracking table 533 is deleted, and the association between ROI number ROI1 and terminal number R1, i.e., the corresponding record, is deleted from the association table 534.

[0105] As described above, the monitoring device 5 to which the information processing device according to the first embodiment is applied uses a tracking unit 511 to detect whether the settlement operation has been completed for each of the multiple transaction processing devices 1 that the customer SC performs settlement operations on, based on image information that can identify the status of each of the multiple transaction processing devices 1. The monitoring device 5 also uses a tracking unit 511 to track the movement of the customer SC, and uses a linking unit 512 to link the customer SC with the transaction processing device 1 that the customer SC performs settlement operations on, based on the location of the tracked customer SC. The monitoring device 5 then uses a fraud detection unit 514 to notify of a payment omission if the status of the transaction processing device 1 linked to the customer SC that has moved to a predetermined location is not that the settlement operation has been completed. Thus, with the monitoring device 5 to which the information processing device according to the first embodiment is applied, by using image information to detect whether or not the transaction processing device 1 has completed the settlement operations, it is no longer necessary to communicate with the transaction processing device 1. When a customer SC moves to a designated location, if the transaction processing device 1 associated with that customer SC has not completed the settlement operations, it notifies the customer of a payment omission, making it easy to notify them of payment omissions. Furthermore, in order to deal with fraudulent payment omissions, gate systems have emerged that install gates at the exit of the checkout area, allowing customers to open the gate by holding a receipt issued by the accounting terminal over the gate upon completion of payment. However, such gate systems are costly and worsen the user experience, as it requires customers to hold a receipt over the gate. Therefore, notification systems have been proposed that communicate with the accounting terminal to obtain its status information, detect the customer's location, and notify store staff if the customer leaves the area where the accounting terminal is installed before completing the accounting process at the terminal. However, in order to connect to the self-checkout system installed in the store and obtain status information from the accounting terminal, knowledge of the information exchanged by that self-checkout system is necessary. Therefore, if knowledge of the checkout system installed in the store is not available or cannot be obtained, it becomes impossible to implement the notification system. In contrast, the monitoring device 5 to which the information processing device according to the first embodiment is applied does not need to communicate with the transaction processing device 1, so it can be applied to the store even if knowledge of the checkout system installed in the store is not available or cannot be obtained.

[0106] In this first embodiment, the monitoring device 5 notifies the fraud detection unit 514 of a payment omission when a customer SC linked to the transaction processing device 1, which is detected as not having completed the settlement operation, moves to a predetermined position. Therefore, according to the monitoring device 5 in the first embodiment, payment failure detection is performed only for customer SCs linked to the transaction processing device 1, rather than for all customer SCs that enter the checkout area CA, thus reducing the processing load. In addition, payment failure detection is not performed for customer SCs and store staff SPs that are not linked to the transaction processing device 1, thus suppressing the occurrence of false notifications. Normally, store staff SPs, who are attendants, pay particular attention to customer SCs that enter the checkout area CA and attempt to leave without operating the transaction processing device 1, so the probability of problems occurring is low even without monitoring by the monitoring device 5.

[0107] Furthermore, in the first embodiment, the monitoring device 5, using the terminal status detection unit 513, detects whether the transaction processing device 1 has completed its settlement operations based on an image of the warning light 111 included in the image information captured by the transaction processing device 1, which is equipped with a warning light 111 that indicates whether the transaction processing device 1 has completed its settlement operations. Therefore, according to the monitoring device 5 in the first embodiment, by determining the status of the transaction processing device 1 displayed by the warning light 111, it is possible to detect the status of the transaction processing device 1 without communicating with the transaction processing device 1.

[0108] Furthermore, the information processing system according to the first embodiment applied to the self-checkout system includes an attendant terminal 3 used by a store employee, a checkout area camera 4 that acquires image information that can identify the status of each of the multiple transaction processing devices 1 in which the customer SC performs settlement operations, and a monitoring device 5 according to the first embodiment that communicates with the attendant terminal 3 and the checkout area camera 4. The monitoring device 5, using a terminal status detection unit 513, detects whether the settlement operations have been completed for each of the multiple transaction processing devices 1 based on the image information acquired by the checkout area camera 4, and if the status of the transaction processing device 1 associated with the customer SC who has moved to a predetermined location is not in a state where the settlement operations have been completed, the monitoring device 5 notifies the attendant terminal 3 of the payment omission. Thus, according to the information processing system of the first embodiment, by using image information to detect whether the transaction processing device 1 has completed the settlement operations, the need to communicate with the transaction processing device 1 is eliminated. Furthermore, if the transaction processing device 1 associated with the customer SC has not completed the settlement operations when the customer SC moves to a designated location, a payment omission is notified, making it easy to notify of payment omissions.

[0109] Furthermore, in the information processing system according to the first embodiment, the monitoring device 5 tracks the movement of the customer shopping center using the tracking unit 511 based on the camera images captured by the checkout area camera 4. Therefore, there is no need to prepare a separate camera from the checkout area camera 4 for tracking the movement of customer shopping centers, and an information processing system can be built at a low cost.

[0110] [Second Embodiment] Next, a second embodiment will be described. Note that the same configurations and operations as in the first embodiment will be denoted by the same reference numerals as in the first embodiment, and their descriptions will be omitted.

[0111] Figure 15 is a flowchart showing an example of the main steps of the payment omission detection process performed by the processor 51 of the monitoring device 5 to which the information processing device according to the second embodiment is applied. The processor 51 performs this process at regular intervals, for example, every few seconds, based on a control program stored in the main memory 52 or auxiliary storage device 53. The procedure shown in Figure 15 is an example. The procedure is not particularly limited as long as similar results can be obtained.

[0112] As ACT251, the terminal status detection unit 513 implemented in the processor 51 initializes the value of the internal counter n to "1". In this embodiment, the value of this internal counter n is used to specify the ROI number.

[0113] As ACT252, the terminal status detection unit 513 obtains the location information of the ROI number n, i.e., the ROIn customer service center. Specifically, the terminal status detection unit 513 reads the ROIn customer service center location information from the tracking table 533.

[0114] As ACT253, the terminal status detection unit 513 determines whether the customer SC has moved to a predetermined location stored in the setting storage unit 531. Specifically, the terminal status detection unit 513 determines whether the current location of the customer SC, indicated by the acquired location information, is outside the predetermined range RR stored in the setting storage unit 531. Alternatively, it may determine whether the current location of the customer SC, indicated by the location information, has crossed a predetermined line. If the customer SC has moved to a predetermined location, the terminal status detection unit 513 determines YES in ACT253 and proceeds to the process of ACT256. If the customer SC has not moved to a predetermined location, the terminal status detection unit 513 determines NO in ACT253 and proceeds to the process of ACT254.

[0115] In ACT254, the terminal status detection unit 513 determines whether the value of the internal counter n is "8", that is, whether it has detected a status for all transaction processing devices 1 provided by the self-checkout system. If the value of the internal counter n is "8", the terminal status detection unit 513 determines YES in ACT254 and terminates this process. If the value of the internal counter n is not "8", the terminal status detection unit 513 determines NO in ACT254 and proceeds to the process of ACT255.

[0116] As ACT255, the terminal status detection unit 513 increments the value of the internal counter n by "+1". After that, the terminal status detection unit 513 proceeds to the process of ACT252.

[0117] As ACT256, the fraud detection unit 514 implemented in the processor 51 checks the terminal number of the transaction processing unit 1 associated with customer SC of ROI number n using the association table 534.

[0118] As ACT257, the fraud detection unit 514 obtains an image of the patrol lamp 111 of the transaction processing device 1 corresponding to the confirmed terminal number from the camera images stored in the camera image temporary storage unit 532 for a certain period of time, and detects the terminal status of the transaction processing device 1 based on the status information of the transaction processing device 1 represented by the color pattern and light emission pattern of the patrol lamp 111.

[0119] As ACT258, the fraud detection unit 514 determines whether the detected terminal is in operation or not. If it is in operation, the fraud detection unit 514 determines YES in ACT258 and proceeds to the process of ACT261. If it is not in operation, i.e., idle, the fraud detection unit 514 determines NO in ACT258 and proceeds to the process of ACT259.

[0120] As ACT259, the fraud detection unit 514 deletes the ROI number n, i.e., ROIn, and its corresponding location information from the tracking table 533. In other words, the fraud detection unit 514 deletes the ROIn record.

[0121] As ACT260, the fraud detection unit 514 deletes the ROI number n, i.e., ROIn, and its corresponding terminal number stored in the linking table 534. In other words, the fraud detection unit 514 deletes the ROIn record. After that, the terminal status detection unit 513 proceeds to the processing of ACT254.

[0122] As ACT261, the fraud detection unit 514 sends an alert notification to the attendant terminal 3 via the communication interface 54. Subsequently, the fraud detection unit 514 proceeds to the processing of ACT254.

[0123] As described above, the monitoring device 5 to which the information processing device according to the second embodiment is applied detects the status of the transaction processing device 1 associated with the customer SC when the tracking customer SC moves to a predetermined location using the terminal status detection unit 513, and the fraud determination unit 514 notifies of a payment omission if the transaction processing device 1 associated with the customer SC that has moved to this predetermined location is not in a state where the settlement operation has been completed. Thus, with the monitoring device 5 to which the information processing device according to the second embodiment is applied, payment failure detection is performed only for the transaction processing device 1 associated with the customer SC that has moved to a specified position, rather than for all of the transaction processing devices 1 installed in the checkout area CA, thus reducing the amount of processing. Furthermore, payment failure detection is not performed for customer SCs or store staff SPs that are not associated with a transaction processing device 1, thus suppressing the occurrence of false notifications. Normally, store staff SPs, who are attendants, pay particular attention to customer SCs that enter the checkout area CA and attempt to leave the checkout area CA without operating a transaction processing device 1, so the probability of problems occurring is low even without monitoring by the monitoring device 5.

[0124] [Third Embodiment] In the monitoring device 5 to which the information processing devices according to the first and second embodiments are applied, the current status of each transaction processing device 1 is determined based on the display state of the warning light 111. This embodiment is an example in which the object of determination used to determine the current status of each transaction processing device 1 is something other than the warning light 111. Specifically, in this embodiment, the display screen of the touch panel 104 of each transaction processing device 1 is used as the object of determination. The following explanation will use the case where the transaction processing device 1 is a self-service POS terminal 100 as an example, but it goes without saying that the transaction processing device 1 may be a self-service accounting terminal 150.

[0125] Figure 16 shows an example of an idle screen 1041 displayed on the touch panel 104 of a self-service POS terminal 100, which is an example of a transaction processing device 1 in the third embodiment. When the self-service POS terminal 100 is available, a predetermined idle screen 1041 indicating this is displayed on the touch panel 104. For example, as shown in Figure 16, the idle screen 1041 includes a display that shows the terminal number of the self-service POS terminal 100 to the customer service center.

[0126] Figure 17 shows an example of a transaction start screen 1042 displayed on the touch panel 104 of the self-service POS terminal 100. For example, when a customer service center touches the touch panel 104, the screen switches from the idle screen 1041 shown in Figure 16 to this transaction start screen 1042.

[0127] Figure 18 shows an example of a registration screen 1043 displayed on the touch panel 104 of a self-service POS terminal 100. The registration screen 1043 displays a list of registered products and their prices.

[0128] Figure 19 shows an example of a payment selection screen 1044 displayed on the touch panel 104 of the self-service POS terminal 100. When the "Proceed to Checkout" button 10431, which is included in the registration screen 1043 shown in Figure 18, is touched, the screen switches to this payment selection screen 1044.

[0129] Figure 20 shows an example of a transaction completion screen 1045 displayed on the touch panel 104 of the self-service POS terminal 100. This transaction completion screen 1045 is displayed when settlement is completed in the transaction processing device 1. After the transaction completion screen 1045 is displayed for a certain period of time, the screen returns to the idle screen 1041 shown in Figure 16.

[0130] Therefore, in this embodiment, an image of the display screen of the touch panel 104 of each transaction processing device 1 is extracted from the camera image captured by the checkout area camera 4, and the current status of the transaction processing device 1 is determined based on what screen is displayed.

[0131] Specifically, the terminal status detection unit 513 of the monitoring device 5 determines that the transaction processing device 1 is in an idle state if the touch panel 104 is displaying the idle screen 1041. If the display on the touch panel 104 switches from the idle screen 1041 to the transaction start screen 1042, the terminal status detection unit 513 determines that the transaction processing device 1 is in operation. If the display on the touch panel 104 is the registration screen 1043, the payment selection screen 1044, or a settlement screen (not shown), the terminal status detection unit 513 determines that the transaction processing device 1 is in operation. Furthermore, if the display on the touch panel 104 becomes the transaction end screen 1045, the terminal status detection unit 513 may determine that the self-POS terminal 100 is in an idle state immediately, or after a certain period of time has elapsed, without waiting for it to switch back to the idle screen 1041.

[0132] Thus, the touch panel 104 is an example of a display component provided by the transaction processing device 1 that indicates whether or not the transaction processing device 1 has completed operations related to settlement. It can also be said that the touch panel 104 is an example of a display component that indicates whether the transaction processing device 1 is idle or in operation using at least one of characters and symbols.

[0133] Specifically, in the monitoring device 5, ACT202 and ACT257 extract the display screen portion of the touch panel 104 from the camera image, rather than the portion of the warning light 111, and detect the status of the transaction processing device 1 based on the display status of that screen.

[0134] In the first and second embodiments, a certain amount of video footage was required to determine whether the patrol lamp 111 was flashing or lit; however, in this embodiment, a single-shot image may suffice.

[0135] As described above, the monitoring device 5 to which the information processing device according to the third embodiment is applied detects whether the transaction processing device 1 has completed the settlement operation based on the terminal status detection unit 513, which includes an image of the display screen of the touch panel 104, which is provided by the transaction processing device 1 and indicates whether the transaction processing device 1 has completed the settlement operation, in the image information captured from the touch panel 104 display screen. Therefore, with the monitoring device 5 to which the information processing device of the third embodiment is applied, the status of the transaction processing device 1 can be determined by taking a picture of the touch panel 104 that displays the status of the transaction processing device 1.

[0136] The touch panel 104 indicates the status of the transaction processing device 1 using at least one of characters and symbols such as buttons. Therefore, according to the monitoring device 5 in the third embodiment, the status of the transaction processing device 1 can be easily determined simply by identifying the characters and / or symbols displayed on the screen of the touch panel 104.

[0137] [Fourth Embodiment] In the third embodiment, the display screen of the touch panel 104 of each transaction processing device 1 is used as the target for determination, but in this embodiment, the display screen of the touch panel provided by the attendant terminal 3 is used as the target for determination.

[0138] Figure 21 is a schematic diagram showing a self-checkout system to which the information processing system according to the fourth embodiment is applied, including a monitoring device 5 to which the information processing device according to the fourth embodiment is applied. Figure 22 is a schematic diagram showing an example of a store checkout area CA in the fourth embodiment.

[0139] In this embodiment, the attendant terminal 3 is connected to a communication network NW and has the function of monitoring multiple transaction processing devices 1. As shown in Figures 21 and 22, the information processing system according to the fourth embodiment includes, in addition to the checkout area camera 4, an attendant screen camera 6 installed so that its imaging range 61 becomes the display screen of the touch panel of the attendant terminal 3. The attendant screen camera 6 transmits the captured image to the monitoring device 5.

[0140] The touch panel display screen of the attendant terminal 3, which has the function of monitoring the transaction processing device 1, displays a monitoring screen 31 as shown in Figure 13. As mentioned above, the attendant terminal 3 displays the current status of each transaction processing device 1 in an identifiable manner by changing the color, density, or shape of the icons displayed in the terminal list area 32 of the monitoring screen 31 based on the status information received from each transaction processing device 1.

[0141] Therefore, in this embodiment, the attendant screen camera 6 captures the monitoring screen 31 displayed on the touch panel 104 of the attendant terminal 3. The terminal status detection unit 513 of the monitoring device 5 then extracts an image from the image captured by the attendant screen camera 6 that shows the status of the transaction processing device 1 to be detected in the terminal list area 32, and determines the current status of the transaction processing device 1 to be detected based on what image is displayed. By fixing the positional relationship between the attendant screen camera 6 and the monitoring screen 31 of the attendant terminal 3, it is possible to know which pixel in the captured image corresponds to which icon of which transaction processing device 1. If the positional relationship is not fixed, the monitoring device 5 can detect which pixel in the captured image corresponds to which icon of which transaction processing device 1 by identifying the terminal list area 32 using known image recognition processing. Of course, the current status of the transaction processing device 1 can be determined by using the image in the terminal status area 33 instead of the image in the terminal list area 32, or the accuracy of the determination can be further improved by using both the image in the terminal list area 32 and the image in the terminal status area 33.

[0142] Thus, the attendant terminal 3 is an example of an attendant terminal that communicates with each transaction processing unit 1 and displays the status of each transaction processing unit 1, and the touch panel of the attendant terminal that displays the monitoring screen 31 is an example of a display screen that displays the status of multiple transaction processing units 1. The attendant screen camera 6 is an example of an acquisition device that acquires image information that can identify the status of each of the multiple transaction processing units 1.

[0143] Specifically, in ACT202 and ACT257, the monitoring device 5 extracts image portions such as icons related to the transaction processing device 1 to be detected from the monitoring screen 31 displayed on the touch panel of the attendant terminal 3, based on the image captured by the attendant screen camera 6, and detects the status of the transaction processing device 1 based on the displayed content of that image portion.

[0144] As described above, the monitoring device 5 to which the information processing device according to the fourth embodiment is applied detects whether or not the transaction processing device 1 has completed the settlement operation based on the terminal status detection unit 513, which captures the images of the status displays for the multiple transaction processing devices 1 included in the image information of the monitoring screen 31 that displays the status of the multiple transaction processing devices 1 on the attendant terminal 3 that communicates with each transaction processing device 1 and displays the status of each transaction processing device 1. Therefore, with the monitoring device 5 to which the information processing device according to the fourth embodiment is applied, the status of the transaction processing device 1 can be easily determined by taking a picture of the monitoring screen 31 of the attendant terminal 3 which displays the status of the transaction processing device 1.

[0145] [Other embodiments] The embodiments of the information processing device have been described above, but the embodiments are not limited thereto.

[0146] For example, in the first to third embodiments, images of the warning lights 111 and touch panels 104 of each transaction processing device 1 are extracted from the camera images of the checkout area camera 4, but each transaction processing device 1 may also be equipped with a camera to photograph the warning lights 111 and touch panels 104.

[0147] Furthermore, the conditions under which each transaction processing device 1 detects that the settlement operation has not been completed based on the camera footage from the checkout area camera 4 are not limited to the conditions described in the embodiment, but may be arbitrarily set by the administrator of the monitoring device 5, which acts as an information processing device.

[0148] Furthermore, the information processing device is not limited to the monitoring device 5; its functions may also be provided by the attendant terminal 3 or the computer constituting the store server 2. Moreover, by connecting the internet connection device to the communication network NW, a server computer located on the cloud may be used as the information processing device.

[0149] In addition, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the invention and its equivalents as described in the claims. [Explanation of Symbols]

[0150] 1...Transaction processing unit, 2...Store server, 3...Attendant terminal, 4...Checkout area camera, 5...Monitoring device, 6...Attendant screen camera, 31...Monitoring screen, 32...Terminal list area, 33...Terminal status area, 34...Popup alert, 35...Customer image, 36..."OK" button, 51...Processor, 511...Tracking unit, 512...Linking unit, 513...Terminal status detection unit, 514...Fraud detection unit, 52...Main memory, 53...Auxiliary storage device, 531...Setting storage unit, 532...Camera image temporary storage unit, 533...Tracking table, 534...Linking table, 54...Communication interface, 55...Input / output device, 56...System transmission path, 100...Self-service POS terminal, 104...Touch panel, 1041...Idle screen, 1042...Transaction start screen, 1043...Registration screen, 10431..."Proceed to checkout" button, 1044...Selection screen, 1045...Transaction completion screen, 111...Patrol light, 112...First light-emitting part, 113...Second light-emitting part, CA...Checkout area, ENT...Entrance gate, EXT...Exit gate, NW...Communication network, RR...Specified range, SC...Customer, SP...Employee.

Claims

1. A detection unit detects whether or not the settlement operation has been completed for each of the multiple transaction processing devices, based on image information that can identify the status of each of the multiple transaction processing devices in which the customer performs settlement operations, A tracking unit for tracking the movement of the aforementioned customer, A linking unit that links the customer and the transaction processing device on which the customer performs the settlement operation based on the location of the customer being tracked, If the status of the transaction processing device associated with the customer who has moved to a predetermined location is not that the settlement operation has been completed, a notification unit notifies of the payment failure. An information processing device equipped with the following features.

2. The information processing device according to claim 1, wherein the notification unit notifies the customer associated with the transaction processing device, which has been detected as not having completed the settlement operation, of the missed payment when the customer moves to the designated location.

3. The detection unit detects the state of the transaction processing device associated with the customer when the customer being tracked moves to the designated location. The information processing apparatus according to claim 1, wherein the notification unit notifies of the payment omission when the transaction processing apparatus linked to the customer that has moved to the specified position is not in a state where the settlement operation has been completed.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the detection unit detects whether the transaction processing apparatus is in a state where the settlement operation has been completed, based on an image of the display member included in the image information obtained by photographing the display member of the transaction processing apparatus, which is provided in the transaction processing apparatus, that indicates whether the transaction processing apparatus is in a state where the settlement operation has been completed.

5. The information processing apparatus according to any one of claims 1 to 3, wherein the detection unit detects whether the transaction processing apparatus is in a state in which the settlement operation has been completed, based on the image of the status display for the plurality of transaction processing apparatuses included in the image information obtained by taking a picture of the display screen that displays the status of the plurality of transaction processing apparatuses on an attendant terminal that communicates with each of the transaction processing apparatuses and displays the status of each of the transaction processing apparatuses.

6. The attendant terminal used by the store staff, An acquisition device that acquires image information that can identify the status of each of the multiple transaction processing devices in which a customer performs settlement operations, An information processing device that communicates with the attendant terminal and the acquisition device, Equipped with, The aforementioned information processing device is A detection unit that, based on the image information acquired by the acquisition device, detects whether or not the settlement operation has been completed for each of the plurality of transaction processing devices, A tracking unit for tracking the movement of the aforementioned customer, A linking unit that links the customer and the transaction processing device on which the customer performs the settlement operation based on the location of the customer being tracked, If the status of the transaction processing device associated with the customer who has moved to a predetermined location is not that the settlement operation has been completed, a notification unit notifies the attendant terminal of the payment omission. An information processing system equipped with the following features.

Images