Information processing system, information processing device and control program thereof

Abstract

To provide an information processing system capable of appropriately coping with a behavior in which a customer leaves a settlement terminal without completing the settlement, an information processing device and its control program.SOLUTION: An information processing device includes operation recognition means, behavior recognition means, and notification means. The operation recognition means recognizes operation to a settlement terminal by an operator. The behavior recognition means recognizes a behavior of the operator performing the operation to the settlement terminal. If the behavior recognition means recognizes a disappearance behavior in which the operator leaves the settlement terminal before the operation recognition means recognizes that the operation to the settlement terminal is settlement end operation, the notification means notifies the disappearance behavior.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing system, an information processing device, and a control program therefor. [Background technology]

[0002] In recent years, self-service POS (Point of Sales) terminals have been gaining attention in supermarkets and other retail stores from the perspective of reducing labor costs and taking measures to prevent the spread of infectious diseases. Self-service POS terminals are fully self-service payment terminals that require customers to perform all operations themselves, from registering the items they wish to purchase to paying. This can lead to situations where customers leave a self-service POS terminal without completing the payment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-084115 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the embodiments of the present invention is to provide an information processing system, an information processing device, and a control program therefor that can appropriately deal with the act of a customer leaving a payment terminal without completing payment. [Means for solving the problem]

[0005] In one embodiment, the information processing device includes an operation recognition means, a behavior recognition means, and a notification means. The operation recognition means recognizes an operation on the payment terminal by an operator. The behavior recognition means recognizes a behavior of the operator performing an operation on the payment terminal. The notification means notifies the operator when the behavior recognition means recognizes a disappearance behavior of the operator moving away from the payment terminal before the operation recognition means recognizes the operation on the payment terminal as a payment completion operation. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal is installed. [Figure 2] FIG. 2 is a diagram for explaining the positional relationship between the self-service POS terminal and the camera. [Figure 3] FIG. 3 is a schematic diagram showing an example of a monitoring image displayed on the display of the attendant terminal. [Figure 4] FIG. 4 is a block diagram showing the main circuit configuration of the information processing device. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of the management table. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of a message table. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of a time-series buffer. [Figure 8] FIG. 8 is a flowchart for explaining the function of the behavior recognition unit. [Figure 9] FIG. 9 is a flowchart for explaining the function of the operation recognition unit. [Figure 10] FIG. 10 is a flowchart for explaining the function of the operation recognition unit. [Figure 11] FIG. 11 is a flowchart for explaining the function of the determination unit. [Figure 12] FIG. 12 is a flowchart for explaining the function of the determination unit. [Figure 13] FIG. 13 is a flowchart for explaining the function of the notification unit. DETAILED DESCRIPTION OF THE INVENTION

[0007] An embodiment will be described below with reference to the drawings. FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal 11 has been introduced. The system includes a self-service POS system 100 and an information processing system 200. The self-service POS system 100 includes a plurality of self-service POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The plurality of self-service POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

[0008] The self-service POS terminal 11 is a fully self-service payment terminal that allows customers to perform operations from registering purchased items to making payments themselves. The customer can also be referred to as an operator. The POS server 12 is a server computer that centrally controls the operations of each self-service POS terminal 11. The display control device 13 is a controller that generates a monitoring image SC (see FIG. 3) for each self-service POS terminal 11 based on data signals output from each self-service POS terminal 11 and displays the image on the display device of the attendant terminal 14. The attendant terminal 14 is a terminal used by store clerks known as attendants to monitor the status of each self-service POS terminal 11. The attendant terminal 14 is equipped with a display such as a liquid crystal display or an organic electroluminescence (EL) display. The attendant terminal 14 divides the display screen into multiple sections, and displays a different monitoring image SC of the attendant terminal 14 for each section. This self-service POS system 100 can be implemented using conventionally known systems as is.

[0009] The information processing system 200 includes a plurality of cameras 21 and an information processing device 22. The plurality of cameras 21 correspond one-to-one to the plurality of self-service POS terminals 11. The cameras 21 are used to capture images of customers operating the corresponding self-service POS terminals 11. The cameras 21 can also be referred to as imaging devices.

[0010] The information processing device 22 has functions as a behavior recognition unit 221, an operation recognition unit 222, a determination unit 223, and an alarm unit 224. The behavior recognition unit 221 has a function of recognizing the behavior of a customer operating the self-service POS terminal 11 based on the image data output from each camera 21. The operations on the self-service POS terminal 11 will be explained later. The behavior recognition unit 221 can also be referred to as behavior recognition means. The operation recognition unit 222 has a function of recognizing an operation by a customer on the self-service POS terminal 11 based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be referred to as operation recognition means. The determination unit 223 has a function of determining whether or not the customer has performed an abnormal behavior based on the customer's behavior with respect to a product for which a product registration operation has been recognized, based on the recognition results by the behavior recognition unit 221 and the operation recognition unit 222. The determination unit 223 can also be referred to as determination means. The notification unit 224 has a function of notifying the attendant of the abnormal behavior when the determination unit 223 determines whether or not the attendant is engaging in abnormal behavior. The notification is made to the attendant from the attendant terminal 14, for example. The notification unit 224 can also be called notification means.

[0011] 2 is a diagram for explaining the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external configuration of the self-service POS terminal 11 will be described.

[0012] The self-service POS terminal 11 comprises a main body 40 placed on the floor and a scale unit 51 placed beside the main body 40. A touch panel 41 is attached to the top of the main body 40. The touch panel 41 is composed of a display and a touch sensor. The display is a device for displaying various screens to the operator, i.e., customer, who operates the self-service POS terminal 11. The touch sensor is a device for detecting touch inputs made by customers on the screen.

[0013] The main body 40 has a basket stand 60 in the center of the side opposite the side where the scale unit 51 is installed. The basket stand 60 is for customers coming from the sales floor to place baskets containing purchased items. Customers work by standing on the front side of the main body 40 in FIG. 2 so that they can see the screen of the touch panel 41. Therefore, from the customer's perspective, the basket stand 60 is on the right side of the main body 40, and the scale unit 51 is on the left side. In the following explanation, the side where the customer stands is referred to as the front of the main body 40, the side where the scale unit 51 is installed is referred to as the left side of the main body 40, and the side where the basket stand 60 is installed is referred to as the right side of the main body 40.

[0014] The main body 40 has a scanner reading window 42, a card insertion slot 43, a receipt issuing slot 44, a coin insertion slot 45, a coin dispensing slot 46, a bill insertion slot 47, and a bill dispensing slot 48 formed on its front side. A communication cable 61 extends from the right side of the main body 40 to the outside, and a reader / writer 62 for electronic money media is connected to the tip of this communication cable 61. The reader / writer 62 is placed on a stand 63 provided on the upper right side of the main body 40.

[0015] A display pole 64 is attached to the top surface of the main body 40. The display pole 64 has a light-emitting unit 65 at its tip that selectively emits light, for example, in blue or red. The display pole 64 displays the status of the self-service POS terminal 11, such as waiting, operating, calling, or error, depending on the color of light emitted by the light-emitting unit 65.

[0016] Scale unit 51 has a structure in which a scale pan 54 is provided on top of housing 50, and bag holder 52 is attached to the top of scale pan 54. The top surface of scale pan 54 serves as a placement surface 55. Bag holder 52 has a pair of holding arms 53, which hold a plastic shopping bag or a shopping bag brought by a customer, known as a "my bag." The scale unit 51 measures the weight of an item placed in a shopping bag or a personal bag held by the holding arm 53 and placed on the placement surface 55.

[0017] Next, the positional relationship between the self-service POS terminal 11 and the camera 21 will be described. As shown in FIG. 2, the camera 21 is installed in a position where it can capture an image of a customer standing in front of the self-service POS terminal 11 and facing the main body 40, the scale unit 51, the basket stand 60, and other components from above.

[0018] A customer standing in front of the self-service POS terminal 11 first places a basket containing purchased items on the basket stand 60 on the right side, and then places a shopping bag or a personal bag on the holding arm 53 on the left side. Next, the customer operates the touch panel 41 according to the guidance displayed on the touch panel 41 to declare that they want to start using the self-service POS terminal 11.

[0019] The customer then picks up each item to purchase from the basket placed on the basket stand 60. If the purchased item has a barcode, the customer registers the item by holding the barcode over the reading window 42 and having it read by the scanner. If the purchased item does not have a barcode, the customer registers the item by operating the touch panel 41 to select the item from a list of items without barcodes. After registering the items, the customer places them in a shopping bag, their own bag, or the like.

[0020] After registering all of the purchased items, the customer operates touch panel 41 to select a payment method. For example, if cash payment is selected, the customer inserts bills or coins into bill slot 47 or coin slot 45, and removes the change from bill dispensing outlet 48 or coin dispensing outlet 46. For example, if electronic money payment is selected, the customer holds the electronic money medium over reader / writer 62. For example, if credit card payment is selected, the customer inserts the credit card into card slot 43. After completing the payment in this way, the customer receives the receipt issued from receipt issuing slot 44 and leaves the store with the shopping bag or personal bag removed from holding arm 53.

[0021] Fig. 3 is a schematic diagram showing an example of a monitoring image SC displayed on the display of the attendant terminal 14. The monitoring image SC is an example of an image that transitions in response to an operation on the self-service POS terminal 11. As described above, the display of the attendant terminal 14 displays separate monitoring images SC for each of the multiple self-service POS terminals 11. Fig. 3 shows an example of a monitoring image SC for one of the self-service POS terminals 11. The configuration of the monitoring image SC for the other self-service POS terminals 11 is similar, so a description thereof will be omitted here.

[0022] As shown in FIG. 3, the monitoring image SC includes a register number column 71, a terminal status column 72, an error information column 73, a declaration information column 74, a details column 75, and a total column .

[0023] The cash register number column 71 is a column for displaying the cash register number. The cash register number is a series of numbers assigned to each self-service POS terminal 11 so as to identify each self-service POS terminal 11 individually. The cash register number is identification information for identifying each self-service POS terminal 11.

[0024] The terminal status field 72 is a field for displaying the operation status of the self-service POS terminal 11. In this embodiment, the terminal status field 72 displays one of the operation statuses: "Waiting," "Start of use," "Registering," "Payment started," and "Payment in progress."

[0025] "Waiting" is the state from when the previous customer has finished paying until the next customer declares that they are ready to use the service. An initial image is displayed on the touch panel 41 of the self-service POS terminal 11 in the "waiting" state. The initial image is, for example, an image including touch buttons that allow the customer to select whether to use the store's provided shopping bags or their own bag. Furthermore, when the operating state of the self-service POS terminal 11 is "starting use," "registering," or "starting payment," if a set time has passed without any operation on the self-service POS terminal 11 being recognized, the operating state of the self-service POS terminal 11 will return to "waiting." The set time is, for example, 30 seconds, 1 minute, etc. The set time can be set as desired.

[0026] "Start of use" is the state in which a customer standing in front of the self-service POS terminal 11 declares the start of use for payment. The customer selects on the initial image whether to use a plastic bag or a personal bag. This selection operation declares the start of use. In response to this selection operation, the operating state of the self-service POS terminal 11 becomes "start of use."

[0027] "Registering" is a state in which the self-service POS terminal 11 is accepting a customer's operation to register a purchased item. When the first purchased item is registered, the operating state of the self-service POS terminal 11 becomes "registering." Thereafter, the operating state of the self-service POS terminal 11 remains "registering" until the transition to payment is declared. In addition, a soft key, for example, "Cancel registration" is displayed on the touch panel 41 of the self-service POS terminal 11 in the "registering" state. A customer who wishes to cancel the registration of a purchased item touches the "Cancel registration" soft key. The operation of a customer touching the "Cancel registration" soft key is an example of a cancel operation that cancels operations on the self-service POS terminal 11. In response to this cancel operation, the operating state of the self-service POS terminal 11 returns to "standby."

[0028] "Payment Start" is the state in which a customer who has finished registering the items they have purchased declares that they want to move on to payment. The touch panel 41 of the self-service POS terminal 11 in the "Registering" state displays a "Checkout" soft key. The customer who has finished registering the items they have purchased touches the "Checkout" soft key. This operation declares that they want to move on to payment. In response to this operation, the operating state of the self-service POS terminal 11 changes to "Payment Start". Furthermore, after "Payment Start", a soft key for example, "Cancel Payment" is displayed on the touch panel 41 of the self-service POS terminal 11. A customer who wants to cancel the payment for the items they have purchased touches the "Cancel Payment" soft key. The operation of a customer touching the "Cancel Payment" soft key is an example of a cancel operation. In response to this cancel operation, the operating state of the self-service POS terminal 11 returns to "Waiting".

[0029] "Payment in progress" is a state in which payment processing such as cash payment, electronic money payment, credit card payment, etc. For example, when a bill or coin is inserted into the bill insertion slot 47 or the coin insertion slot 45, the operating state of the self-service POS terminal 11 becomes "payment in progress." Then, when the payment processing is completed, the operating state of the self-service POS terminal 11 returns to "standby."

[0030] The error information field 73 is a field for displaying error information that has occurred in the self-service POS terminal 11. The error information is, for example, a communication error, an out-of-receipt error, etc. The declaration information field 74 is a field for displaying the contents of the customer's declaration operation. For example, if the customer selects "my bag," "no bag needed" is displayed, indicating that a plastic bag is not required.

[0031] The details column 75 is a column for displaying detailed information of purchased items registered at the self-service POS terminal 11. The detailed information includes, for example, the product name, number of items, and price of the purchased items. The total column 76 is a column for displaying total information of purchased items registered at the self-service POS terminal 11. The total information includes, for example, the total number of items, the total price, the amount inserted, and the change.

[0032] It should be noted that the configuration of the monitoring image SC is not limited to this. Columns for displaying other items may be arranged. Furthermore, the items of text data displayed in FIG. 3 are not limited to this. Text data of other items may be displayed.

[0033] 4 is a block diagram showing the main circuit configuration of the information processing device 22. The information processing device 22 includes a processor 81, a main memory 82, an auxiliary storage device 83, a clock 84, a timer 85, a camera interface 86, a communication interface 87, and a system bus 88. The system bus 88 includes an address bus, a data bus, etc. The information processing device 22 configures a computer by connecting the processor 81, the main memory 82, the auxiliary storage device 83, the clock 84, the timer 85, the camera interface 86, and the communication interface 87 via the system bus 88.

[0034] The processor 81 corresponds to the central part of the computer. The processor 81 controls each part to realize various functions of the information processing device 22 in accordance with an operating system or an application program. The processor 81 is, for example, a CPU (Central Processing Unit).

[0035] The main memory 82 corresponds to the main storage portion of the computer. The main memory 82 includes a nonvolatile memory area and a volatile memory area. The main memory 82 stores an operating system or application programs in the nonvolatile memory area. The main memory 82 stores data required for the processor 81 to execute processes for controlling each part in the volatile memory area. This type of data may also be stored in the nonvolatile memory area. The main memory 82 uses the volatile memory area as a work area where data is rewritten by the processor 81 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0036] The auxiliary storage device 83 corresponds to the auxiliary storage portion of the computer. As the auxiliary storage device 83, for example, a well-known storage device such as an SSD (Solid State Drive), an HDD (Hard Disc Drive), or an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory) is used alone or in combination. The auxiliary storage device 83 stores data used by the processor 81 when performing various processes, data generated by the processes in the processor 81, etc. The auxiliary storage device 83 may also store application programs.

[0037] The application programs stored in the main memory 82 or the auxiliary storage device 83 include a control program, which will be described later. There are no particular restrictions on the method for installing the control program into the main memory 82 or the auxiliary storage device 83. The control program can be recorded on a removable recording medium, or can be distributed by communication via a network and installed in the main memory 82 or the auxiliary storage device 83. The recording medium can be in any form, such as a CD-ROM or memory card, as long as it can store the program and is readable by the device.

[0038] The clock 84 functions as a time information source for the information processing device 22. The processor 81 obtains the current date and time based on the time information kept by the clock 84.

[0039] The timer 85 counts a set time under the control of the processor 81 .

[0040] The camera interface 86 is an interface for communicating with each camera 21. The imaging data output from each camera 21 is taken into the information processing device 22 via the camera interface 86. The imaging data is data obtained by capturing an image of a customer operating the self-service POS terminal 11 corresponding to the camera 21.

[0041] The communication interface 87 is an interface for communicating with the display control device 13. Image data output from the display control device 13 is taken into the information processing device 22 via the communication interface 87. The image data is data of the monitoring image SC generated for each self-service POS terminal 11.

[0042] The information processing device 22 configured as described above uses a part of the volatile memory area in the main memory 82 as an area for a management table 821 (see FIG. 5) and a message table 822 (see FIG. 6). The information processing device 22 then creates the management table 821 and the message table 822 in this area.

[0043] Fig. 5 is a schematic diagram showing an example of the management table 821. As shown in Fig. 5, the management table 821 has an area for writing one or more tracking IDs in association with the register number of the self-checkout POS terminal 11. The tracking ID is identification information for temporarily identifying a person as a tracking target when it is detected that the person is standing in front of the self-checkout POS terminal 11. Note that the management table 821 is not limited to the data of the above-mentioned items.

[0044] Fig. 6 is a schematic diagram showing an example of the message table 822. As shown in Fig. 6, the message table 822 is a data table that describes the text data of messages in association with message codes MC. The message codes MC and the text data of messages associated therewith will be made clear in the explanation below.

[0045] Furthermore, the information processing device 22 allocates a part of the volatile memory area in the main memory 82 as an area for a time-series buffer 823 (see FIG. 7). The information processing device 22 then creates the same number of time-series buffers 823 as the number of self-checkout POS terminals 11 in this area.

[0046] Fig. 7 is a schematic diagram showing an example of the time-series buffer 823. As shown in Fig. 7, the time-series buffer 823 has areas for writing a tracking ID, a time TM, and a status ST or a message code MC for each register number that identifies the self-service POS terminal 11. The time-series buffer 823 writes the status ST or the message code MC in order of the earliest time TM. The status ST will be explained later.

[0047] The information processing device 22 realizes the functions of the behavior recognition unit 221, the operation recognition unit 222, the determination unit 223, and the notification unit 224 described above by the processor 81 and a control program that controls the processor 81.

[0048] The functions of the behavior recognition unit 221, the operation recognition unit 222, the determination unit 223, and the notification unit 224 are functions provided for each self-service POS terminal 11. Therefore, the functions of the behavior recognition unit 221, the operation recognition unit 222, the determination unit 223, and the notification unit 224 for one self-service POS terminal 11 will be described in detail below. The functions of the behavior recognition unit 221, the operation recognition unit 222, the determination unit 223, and the notification unit 224 for other self-service POS terminals 11 are similar, so the description here will be omitted.

[0049] FIG. 8 is a flowchart for explaining the function of the behavior recognition unit 221. The processor 81 waits to recognize a customer's appearance behavior as ACT1. The appearance behavior is the behavior of a customer appearing at the self-service POS terminal 11. When the processor 81 detects from the image data captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, it recognizes that an appearance behavior has occurred. Examples of cases in which a customer is detected from the image data captured by the camera 21 include when a customer performs operations from registering purchased items to making a payment at the self-service POS terminal 11, or when a customer who has gone to the sales floor to retrieve an item they forgot to buy returns to the self-service POS terminal 11.

[0050] When the customer's appearance behavior is recognized, the processor 81 determines YES in ACT 1 and proceeds to ACT 2. In ACT 2, the processor 81 assigns a tracking ID to the customer.

[0051] The processor 81 acquires the register number of the self-service POS terminal 11 in ACT3. Each camera 21 corresponds one-to-one with each self-service POS terminal 11. Therefore, the processor 81 identifies the self-service POS terminal 11 from the identification information of the camera 21 that captures an image of a customer standing in front of the self-service POS terminal 11, and acquires the register number of that self-service POS terminal 11.

[0052] In ACT4, the processor 81 writes the tracking ID assigned in the processing of ACT2 in association with the register number of the self-checkout POS terminal 11 acquired in the processing of ACT3 in the management table 821 of the main memory 82.

[0053] The processor 81 sets the status ST to "11" in ACT5. That is, when the behavior recognition unit 221 recognizes an appearance behavior toward the self-checkout POS terminal 11, the processor 81 sets the status ST to "11". The processor 81 acquires the current time TM measured by the clock 84 in ACT6. The processor 81 associates the tracking ID assigned in ACT2 with the time TM and the status ST in ACT7 in the time series buffer 823 in which the register number acquired in ACT3 is set.

[0054] The processor 81 waits to recognize the disappearance behavior of a customer as ACT8. The disappearance behavior is the behavior of a customer leaving the self-service POS terminal 11. When the processor 81 can no longer detect the customer from the image data captured by the camera 21, the processor 81 recognizes that the disappearance behavior has occurred. Cases in which the customer can no longer be detected from the image data captured by the camera 21 include, for example, when a customer leaves the self-service POS terminal 11 with purchased items that have not yet been paid for, when a customer who has completed payment leaves the self-service POS terminal 11 to leave the store, when a customer leaves the self-service POS terminal 11 to go to the sales floor to retrieve items they left behind, and when a customer who performed a cancellation operation leaves the self-service POS terminal 11.

[0055] When the disappearance behavior is recognized, the processor 81 determines YES in ACT8 and proceeds to ACT9. The processor 81 sets the status ST to "12" in ACT9. That is, when the disappearance behavior toward the self POS terminal 11 is recognized by the function of the behavior recognition unit 221, the status ST is set to "12".

[0056] In ACT10, the processor 81 acquires the current time TM measured by the clock 84. In ACT11, the processor 81 associates the tracking ID assigned in the processing of ACT2, the time TM, and the status ST and writes them in the time-series buffer 823, to which the register number acquired in the processing of ACT3 has been set. The processor 81 then returns to ACT1. That is, when it is again detected from the image data of the camera 21 that a person is standing in front of the self-service POS terminal 11, the processor 81 executes the processing of ACT2 to ACT11 in the same manner as described above. This concludes the description of the function of the behavior recognition unit 221.

[0057] FIG. 9 is a flowchart for explaining the function of the operation recognition unit 222. The processor 81 waits for the start of use to be declared to the self-service POS terminal 11 as ACT21. When the start of use is declared, "Start of use" is displayed in the terminal status field 72 of the monitoring image SC corresponding to the self-service POS terminal 11. The processor 81 checks whether it can recognize the words "Start of use" from the terminal status field 72 of the monitoring image SC acquired via the display control device 13. If the words "Start of use" can be recognized, the processor 81 recognizes that the start of use has been declared.

[0058] When processor 81 recognizes that the start of use has been declared, it determines YES in ACT21 and proceeds to ACT22. In ACT22, processor 81 acquires the register number of self-checkout POS terminal 11. The register number is displayed in register number column 71 of monitoring image SC. Processor 81 recognizes the characters of the register number from register number column 71 of monitoring image SC acquired via display control device 13, and acquires the characters as the register number.

[0059] In ACT23, the processor 81 sets the status ST to "21." In ACT24, the processor 81 obtains the current time TM measured by the clock 84.

[0060] In ACT25, processor 81 refers to management table 821 to obtain a tracking ID. Specifically, processor 81 refers to management table 821 to obtain the latest tracking ID written in association with the register number obtained in ACT22. If there is one tracking ID associated with the register number, the latest tracking ID is that tracking ID. If there are multiple tracking IDs associated with the register number, the latest tracking ID is the multipleth tracking ID. Note that the specific processing in ACT38, ACT42, and ACT47 in Figure 10, which will be described later, is the same as the processing in ACT25.

[0061] In ACT26, the processor 81 writes the tracking ID, time TM, and status ST obtained in the processing of ACT25 in association with each other in the time-series buffer 823 in which the register number obtained in the processing of ACT22 has been set.

[0062] Therefore, when a customer standing in front of the self-service POS terminal 11 makes a declaration to start using the service, the status ST "21" is first written in the time series buffer 823 corresponding to the self-service POS terminal 11, along with the time TM and tracking ID.

[0063] As ACT27, the processor 81 starts recognizing an operation on the self-checkout POS terminal 11. Specifically, the processor 81 recognizes a product registration operation, a payment start operation, a payment cancellation operation, or a payment completion operation from the transition of information obtained by character recognition of the monitoring image SC acquired via the display control device 13.

[0064] For example, processor 81 recognizes that a product registration operation has been performed when detail information such as the product name, quantity, and price of the purchased product is added to detail column 75. For example, processor 81 recognizes that a payment start operation has been performed when the display in terminal status column 72 switches to "payment start". For example, processor 81 recognizes that a cancellation operation has been performed when the display in terminal status column 72 switches from "registering" to "waiting". For example, processor 81 recognizes that a cancellation operation has been performed when the display in terminal status column 72 switches from "payment start" to "waiting". For example, processor 81 recognizes that a payment completion operation has been performed when the display in terminal status column 72 switches from "payment in progress" to "waiting".

[0065] The processor 81 waits for the recognition of a product registration operation, a payment start operation, a payment cancellation operation, or a payment completion operation as ACT31 to ACT34 in FIG.

[0066] In the standby state of ACT31 to ACT34, when the processor 81 recognizes a product registration operation, it determines YES in ACT31 and proceeds to ACT35. The processor 81 sets the status ST to "22" in ACT35. That is, when a product registration operation on the self-service POS terminal 11 is recognized by the function of the operation recognition unit 222, the status ST is set to "22".

[0067] In the standby state of ACT31 to ACT34, when the processor 81 recognizes a payment start operation, it determines YES in ACT32 and proceeds to ACT36. The processor 81 sets the status ST to "23" in ACT36. That is, when a payment start operation on the self-service POS terminal 11 is recognized by the function of the operation recognition unit 222, the status ST is set to "23".

[0068] When the processor 81 finishes the processing of ACT 35 or ACT 36, the processor 81 proceeds to ACT 37. In ACT 37, the processor 81 obtains the current time TM kept by the clock 84.

[0069] In ACT 38, the processor 81 refers to the management table 821 and acquires the tracking ID. In ACT 39, the processor 81 associates the tracking ID, time TM, and status ST acquired in the processing of ACT 38 and writes them in the time series buffer 823, to which the register number acquired in the processing of ACT 22 in Figure 9 has been set. The processor 81 then returns to the standby state of ACT 31 to ACT 34.

[0070] In the standby state of ACT31 to ACT34, when the processor 81 recognizes a cancel operation, it determines YES in ACT33 and proceeds to ACT40. The processor 81 sets the status ST to "24" in ACT40. That is, when a cancel operation on the self POS terminal 11 is recognized by the function of the operation recognition unit 222, the status ST is set to "24".

[0071] In ACT41, processor 81 obtains the current time TM measured by clock 84. In ACT42, processor 81 references management table 821 to obtain a tracking ID. In ACT43, processor 81 associates and writes the tracking ID, time TM, and status ST obtained in the processing of ACT42 in time series buffer 823, to which the register number obtained in the processing of ACT22 in Figure 9 has been set. With this, processor 81 terminates its function as operation recognition unit 222.

[0072] In the standby state of ACT31 to ACT34, if the processor 81 recognizes a payment completion operation, it determines YES in ACT34 and proceeds to ACT 44. The processor 81 ends the operation recognition for the self-checkout POS terminal 11 in ACT44.

[0073] After completing the operation recognition, the processor 81 sets the status ST to "25" as ACT 45. That is, when the operation recognition unit 222 recognizes a payment completion operation on the self-checkout POS terminal 11, the status ST is set to "25".

[0074] In ACT46, the processor 81 obtains the current time TM measured by the clock 84. In ACT47, the processor 81 references the management table 821 to obtain the tracking ID. In ACT48, the processor 81 associates the tracking ID, time TM, and status ST obtained in the processing of ACT47 and writes them in the time series buffer 823, to which the register number obtained in the processing of ACT22 in Figure 9 has been set. With this, the processor 81 ends its function as the operation recognition unit 222.

[0075] Typically, a customer performs operations on the self-service POS terminal 11 in the following order: start use operation, product registration operation, start payment operation, and end payment operation. Therefore, the status ST is written in the time-series buffer 823 in the order of "21", "22", "23", and "25".

[0076] A customer who wishes to cancel the registration of a purchased item performs the following operations in order on the self-service POS terminal 11: start use operation, product registration operation, and cancellation operation. Therefore, the status ST is written in the time-series buffer 823 in the order of "21", "22", and "24".

[0077] A customer who wishes to cancel payment for a purchased item performs the following operations in the self-service POS terminal 11: start use operation, product registration operation, start payment operation, and cancel operation. Therefore, the status ST is written in chronological order in the time-series buffer 823 as "21", "22", "23", and "24".

[0078] This concludes the description of the function of the operation recognition unit 222. After that, when the operation to start use of the self-service POS terminal 11 is detected again from the data of the monitoring image SC, the processor 81 executes the processes of ACT22 to ACT27 in Fig. 9 and ACT31 to ACT48 in Fig. 10 in the same manner as described above.

[0079] 11 and 12 are flow charts for explaining the function of the determination unit 223. FIG. In ACT51, the processor 81 checks whether or not an emerging behavior has been recognized by the behavior recognition unit 221. When an emerging behavior is recognized, "11" is written as the status ST in the time series buffer 823. When "11" is written as the status ST in the time series buffer 823, the processor 81 determines YES in ACT51 and proceeds to ACT52.

[0080] In ACT52, the processor 81 checks whether the start-of-use operation has been recognized by the operation recognition unit 222. When the start-of-use operation is performed on the self-service POS terminal 11, "21" is written as the status ST in the time-series buffer 823, in which the register number of the self-service POS terminal 11 on which the start-of-use operation was performed is set. If "21" is written as the status ST in the time-series buffer 823, the processor 81 determines YES in ACT52 and proceeds to ACT53.

[0081] In ACT53, the processor 81 checks whether or not a disappearance behavior has been recognized by the behavior recognition unit 221. When a disappearance behavior is recognized, "12" is written as the status ST in the time series buffer 823. Therefore, if "12" is written as the status ST in the time series buffer 823, the processor 81 determines YES in ACT53 and proceeds to ACT54.

[0082] The processor 81 checks whether the disappearance behavior is after a cancel operation as ACT54. When a cancel operation is performed on the self POS terminal 11, "24" is written as the status ST in the time series buffer 823. Therefore, if "24" is written as the status ST in the time series buffer 823 in association with the time TM that is one time before the time TM at which "12" was written as the status ST, the processor 81 determines that the disappearance behavior is after a cancel operation.

[0083] If the disappearance behavior does not occur after a cancel operation, the processor 81 determines NO in ACT54 and proceeds to ACT55. In ACT55, the processor 81 checks whether the disappearance behavior occurs after a product registration operation. When a product registration operation is performed on the self-service POS terminal 11, "22" is written as the status ST in the time series buffer 823. Therefore, if "22" is written as the status ST in the time series buffer 823 in association with the time TM that is one time earlier than the time TM at which "12" was written as the status ST, the processor 81 determines that the disappearance behavior occurred after a product registration operation.

[0084] If the disappearance behavior did not occur after the product registration operation, the processor 81 determines NO in ACT55 and proceeds to ACT56. In ACT56, the processor 81 checks whether the disappearance behavior occurred after the payment start operation. When a payment start operation is performed on the self-service POS terminal 11, "23" is written as the status ST in the time series buffer 823. Therefore, if "23" is written as the status ST in the time series buffer 823 in association with the time TM that is one time earlier than the time TM at which "12" was written as the status ST, the processor 81 determines that the disappearance behavior occurred after the payment start operation.

[0085] If the disappearance behavior does not occur after the payment start operation, i.e., if the disappearance behavior occurs after the payment end operation, the processor 81 determines NO in ACT56. When the customer's disappearance behavior is recognized at the self-service POS terminal 11 where the payment end operation is performed, it may be that the customer has finished paying and left the self-service POS terminal 11 to leave the store. It is determined that the customer's behavior does not correspond to abnormal behavior. With this, the processor 81 ends its function as the determination unit 223.

[0086] In the standby state of ACT54 to ACT56, if the disappearance behavior is after the cancel operation, the processor 81 determines YES in ACT54. When the customer's disappearance behavior is recognized at the self-service POS terminal 11 where the cancel operation was performed, it is possible that the customer touched the [Cancel Registration] soft key displayed on the touch panel 41 to cancel the registration of the purchased item or the [Cancel Payment] soft key to cancel the payment for the purchased item and then left the self-service POS terminal 11. It is determined that this customer's behavior does not constitute abnormal behavior. With this, the processor 81 terminates its function as the determination unit 223.

[0087] In the standby state of ACT54 to ACT56, if the disappearance behavior is after the product registration operation, the processor 81 determines YES in ACT55 and proceeds to ACT 57. In ACT57, the processor 81 sets the message code MC to "91".

[0088] When the disappearance of a customer is recognized at the self-service POS terminal 11 where the product registration operation has been performed, for example, the customer may have left the self-service POS terminal 11 with the purchased products without completing the payment. This customer's behavior is determined to be abnormal. "91" is written as the message code MC in the time-series buffer 823 where the register number of the self-service POS terminal 11 is set. The message code MC "91" is a code that identifies "disappearance of a customer during product registration operation."

[0089] In the standby state of ACT54 to ACT56, if the disappearance behavior is after the payment start operation, the processor 81 determines YES in ACT56 and proceeds to ACT 58. In ACT58, the processor 81 sets the message code MC to "92".

[0090] When the disappearance of a customer is recognized at the self-service POS terminal 11 where the payment initiation operation has been performed, for example, the customer may have left the self-service POS terminal 11 with the purchased items without completing the payment. This customer's behavior is determined to be abnormal. "92" is written as the message code MC in the time-series buffer 823 where the register number of that self-service POS terminal 11 is set. The message code MC "92" is a code that identifies "disappearance of a customer during payment operation."

[0091] When the processor 81 completes the processing of ACT 57 or ACT 58, the processor 81 proceeds to ACT 59. In ACT 59, the processor 81 obtains the current time TM measured by the clock 84.

[0092] In ACT60, the processor 81 refers to the management table 821 to obtain a tracking ID. Specifically, the processor 81 refers to the management table 821 to obtain the latest tracking ID written in association with the register number of the self-checkout POS terminal 11 where the start-of-use operation was recognized. If there is one tracking ID associated with the register number, the latest tracking ID is that tracking ID. If there are multiple tracking IDs associated with the register number, the latest tracking ID is the multipleth tracking ID. Note that the specific processing of ACT76 in FIG. 12, which will be described later, is the same as the processing of ACT60.

[0093] The processor 81 writes the tracing ID, time TM, and message code MC acquired in the processing of ACT60 in the time-series buffer 823 in association with each other as ACT61.

[0094] The processor 81 checks whether or not an appearance behavior has been recognized by the behavior recognition unit 221 as ACT71 in Fig. 12. When an appearance behavior is recognized, "11" is written as the status ST in the time series buffer 823. Therefore, when "12" is written as the status ST in the time series buffer 823 in association with the time TM that is one time before the time TM at which "11" was written as the status ST, the processor 81 determines that the behavior is an appearance behavior after a disappearance behavior.

[0095] As ACT72, the processor 81 calculates the time difference between the time TM when "11" is written as the status ST and the time TM immediately before that time TM when "12" is written as the status ST. That is, the processor 81 calculates the time difference from the time TM when the disappearance behavior is recognized by the behavior recognition unit 221 to the time TM when the appearance behavior is recognized.

[0096] The processor 81 checks whether the time difference is within a predetermined time as ACT 73. The predetermined time is, for example, 30 seconds, 1 minute, etc. The predetermined time can be set arbitrarily.

[0097] If the time difference is not within the predetermined time, that is, if the time difference is outside the predetermined time, the processor 81 determines NO in ACT 73 and ends the function as the determination unit 223.

[0098] If the time difference is within the predetermined time, the processor 81 determines YES in ACT 73 and proceeds to ACT 74. In ACT 74, the processor 81 sets the message code MC to "93".

[0099] When the self-service POS terminal 11 recognizes the customer's disappearance behavior and then recognizes the customer's appearance behavior within a predetermined time, for example, when the customer leaves the self-service POS terminal 11 to go to the sales floor to retrieve an item they left behind and then returns to the self-service POS terminal 11, it is possible that a different customer than the one who left the self-service POS terminal 11 with the unpaid purchased items performs operations at the self-service POS terminal 11 from registering the purchased items to paying for them. The message code MC "93" is stored in the time-series buffer 823 in which the register number of that self-service POS terminal 11 is set. The message code MC "93" is a code that identifies "customer appearance."

[0100] In ACT 75, the processor 81 obtains the current time TM measured by the clock 84. In ACT 76, the processor 81 references the management table 821 to obtain a tracing ID. In ACT 77, the processor 81 associates the tracing ID, time TM, and message code MC obtained in the processing of ACT 76 and writes them in the time series buffer 823. The processor 81 then returns to ACT 53 in Figure 11.

[0101] This concludes the description of the function of the determination unit 223. After that, when an emerging behavior is recognized again, the processor 81 executes the processes of ACT52 to ACT61 in Fig. 11 and ACT71 to ACT77 in Fig. 12 in the same manner as described above.

[0102] FIG. 13 is a flowchart for explaining the function of the notification unit 224. In ACT 81, the processor 81 waits for the message code MC to be written in the time series buffer 823. When the message code MC is written in the time series buffer 823, the processor 81 determines YES in ACT 81 and proceeds to ACT 82. In ACT 82, the processor 81 checks whether the message code MC is "91".

[0103] If the message code MC is "91", the processor 81 determines YES in ACT82 and proceeds to ACT84. In ACT84, the processor 81 issues a notification that "a customer has disappeared during a product registration operation". Specifically, the processor 81 searches the message table 822 to obtain message data with a message code MC of "91". The processor 81 also obtains the register number from the time-series buffer 823 in which "91" is written as the message code MC. The processor 81 then outputs a notification command with the message data and the register number added from the communication interface 87 to the display control device 13.

[0104] The display control device 13 displays the text of the message data on the monitoring image SC of the attendant terminal 14 identified by the register number included in the notification command. As a result, for example, text such as "A customer has disappeared while registering a product at register No. X" is displayed on the monitoring image SC.

[0105] If the message code MC is not "91", the processor 81 determines NO in ACT 82 and proceeds to ACT 83. In ACT 83, the processor 81 checks whether the message code MC is "92".

[0106] If the message code MC is "92", the processor 81 determines YES in ACT83 and proceeds to ACT84. In ACT84, the processor 81 issues a notification that "the customer has disappeared during the payment operation". Specifically, the processor 81 searches the message table 822 to obtain message data with a message code MC of "92". The processor 81 also obtains the register number from the time-series buffer 823 in which "92" is written as the message code MC. The processor 81 then outputs a notification command with the message data and the register number added from the communication interface 87 to the display control device 13.

[0107] The display control device 13 displays the text of the message data on the monitoring image SC of the attendant terminal 14 identified by the register number included in the notification command. As a result, for example, text such as "A customer has disappeared during payment at register No. X" is displayed on the monitoring image SC.

[0108] In this way, when the message code MC is "91" or "92," the notification unit 224 notifies the user that the abnormal behavior has occurred, such as "a customer has disappeared during product registration" or "a customer has disappeared during payment." The attendant may take action such as rushing to the self-service POS terminal 11 with register number X set to check the situation or searching for the customer who has engaged in the abnormal behavior.

[0109] If the message code MC is not "92," that is, if the message code MC is "93," the processor 81 determines NO in ACT83 and proceeds to ACT85. The processor 81 issues a notification of "a customer's arrival" in ACT85. Specifically, the processor 81 searches the message table 822 to obtain message data with a message code MC of "93." The processor 81 also obtains the register number from the time-series buffer 823 in which "93" is written as the message code MC. The processor 81 then outputs a notification command with the message data and the register number added from the communication interface 87 to the display control device 13.

[0110] The display control device 13 displays the text of the message data on the monitoring image SC of the attendant terminal 14 identified by the register number included in the notification command. As a result, for example, text such as "A customer has arrived at register No. X" is displayed on the monitoring image SC.

[0111] In this way, by the function of the notification unit 224, a notification of "a customer has appeared" is made when "93" is written as the message code MC in the time series buffer 823. The attendant will check that a customer has appeared at the self-service POS terminal 11 to which register No. X is set, go to the sales floor to pick up any products that the customer left behind, and check whether the customer has returned to the self-service POS terminal 11, for example.

[0112] After completing the processing of ACT 84 or ACT 85, the processor 81 returns to ACT 81. That is, when the message code MC is written in the time series buffer 823, the processor 81 executes the processing of ACT 82 to ACT 85 in the same manner as described above. This concludes the description of the function of the notification unit 224.

[0113] As described above in detail, the information processing device 22, as the behavior recognition unit 221, recognizes the behavior of a customer operating the self-service POS terminal 11 based on the image data captured by the camera 21. The information processing device 22, as the operation recognition unit 222, recognizes the operation by a customer on the self-service POS terminal 11. The operation by a customer on the self-service POS terminal 11 is recognized based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14. The information processing device 22, as the determination unit 223, determines whether or not the customer has engaged in abnormal behavior from the customer's behavior with respect to a product for which a product registration operation has been recognized, based on the recognition results by the behavior recognition unit 221 and the operation recognition unit 222. The information processing device 22, as the notification unit 224, notifies the customer of the abnormal behavior when the determination unit 223 determines whether or not the abnormal behavior is present.

[0114] For example, when the self-service POS terminal 11 recognizes the disappearance of a customer after the cancellation operation or the payment completion operation is performed, the behavior of the customer is determined not to be abnormal. In this case, the attendant terminal 14 is not used to notify the attendant of the disappearance of the customer. On the other hand, if the disappearance of a customer is recognized at the self-service POS terminal 11 where a product registration operation or a payment start operation is performed without a cancel operation, the customer's behavior is determined to be abnormal. In this case, the attendant is notified via the attendant terminal 14 that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." Therefore, the attendant can rush to the self-service POS terminal 11 to check the situation, search for the customer who has engaged in abnormal behavior, and take other measures.

[0115] For example, if the customer's appearance is recognized at the self-service POS terminal 11 where the customer's disappearance was recognized within a predetermined time after the attendant was notified that "a customer has disappeared during product registration" or "a customer has disappeared during payment," the attendant is notified of "a customer's appearance" using the attendant terminal 14. Therefore, the attendant can check the customer who reappears at the self-service POS terminal 11, go to the sales floor to retrieve the forgotten item, and check whether the customer has returned to the self-service POS terminal 11, or take other measures.

[0116] In this way, according to this embodiment, it is possible to appropriately deal with the behavior of a customer leaving the self-checkout POS terminal 11 without completing payment.

[0117] Although the embodiments of the information processing system, the information processing device, and the control program therefor have been described above, the embodiments are not limited to these.

[0118] In the above embodiment, an example has been given in which one camera 21 is provided for one self-service POS terminal 11. A camera 21 does not necessarily have to be provided for each self-service POS terminal 11. For example, if one camera 21 can capture an image of a customer operating two adjacent self-service POS terminals 11, the number of cameras 21 may be reduced. In that case, however, in ACT 3 of FIG. 8, the register number of the self-service POS terminal 11 closest to the position of the person captured in the captured image data is acquired. Also, for example, multiple cameras 21 may be provided for one self-service POS terminal 11. By doing so, blind spots of the self-service POS terminal 11 can be reduced and customer behavior can be detected with high accuracy.

[0119] In the above embodiment, when a customer's disappearance is recognized at a self-service POS terminal 11 where a product registration operation or a payment start operation has been performed without a cancel operation being performed, an example is given in which the attendant terminal 14 is used to notify the attendant that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." For example, if a customer who has performed a product registration operation or a payment start operation without performing a cancel operation leaves the self-service POS terminal 11 with the purchased product in hand, the attendant may be notified that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." In other words, if a customer leaves the self-service POS terminal 11 without holding the purchased product in hand, the attendant does not need to be notified that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." In this case, the processor 81 estimates the person, i.e., the customer's skeleton, from image data captured by the camera 21, for example. Skeleton estimation can be achieved even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the customer's movements as they hold the purchased product, for example, from hand movements obtained by skeleton estimation. For example, if a customer who has performed a product registration operation or a payment start operation without performing a cancel operation leaves the self-service POS terminal 11, and if a decrease in the total weight of the purchased products measured by the scale unit 51 is detected, the attendant may be notified that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." In other words, if the total weight of the purchased products measured by the scale unit 51 does not change and the customer leaves the self-service POS terminal 11, it is not necessary to notify that "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation."

[0120] In the above embodiment, when a customer's appearance behavior is recognized, the processor 81 assigns a tracking ID to the customer. For example, the processor 81 may identify the customer based on the clothes the customer is wearing from the image data captured by the camera 21. For example, if a customer leaves the self-service POS terminal 11 to retrieve a forgotten item from the sales floor and then returns to the self-service POS terminal 11, the customer who left the self-service POS terminal 11 and the customer who returned are the same person, and therefore the clothes are the same. In this case, even if the processor 81 recognizes the customer's appearance behavior, it does not assign a new tracking ID, assuming that the customer is the same person. For example, if a customer leaves the self-service POS terminal 11 with unpaid purchased items, and then a different customer performs operations at the self-service POS terminal 11 from registering the purchased items to paying for them, the clothes will be different. In this case, the processor 81 assigns a tracking ID to the customer, assuming that the customer is a different person.

[0121] In the above embodiment, the notification unit 224 notifies the attendant via the attendant terminal 14. The notification destination is not limited to the attendant terminal 14. For example, the notification destination may be the self-service POS terminal 11 where the abnormal behavior occurred. In this case, the light-emitting unit 65 may emit light in a predetermined color to notify the store clerk that a disappearing behavior is occurring. Alternatively, the store clerk may be notified that a disappearing behavior is occurring by displaying a message on the touch panel 41 or by voice.

[0122] In the above embodiment, the attendant terminal 14 may incorporate the function of the display control device 13. In this case, the operation recognition unit 222 acquires data of the monitoring image SC from the attendant terminal 14 and recognizes the operation by the customer on the self-service POS terminal 11. Alternatively, the operation recognition unit 222 may acquire a data signal output from each self-service POS terminal 11 from the communication network 15 via a router, for example, and recognize the operation by the customer on the self-service POS terminal 11 based on the data signal.

[0123] In the above embodiment, the self-service POS terminal 11 is exemplified. However, the POS terminal may be a semi-self-service type in which, for example, a store clerk registers purchased items and the customer makes the payment. In this case, the message table 822 contains message codes MC "92" and "93." In other words, the message table 822 does not contain message code MC "91" which identifies "a customer disappears during product registration."

[0124] In the above embodiment, the information processing device 22 may include a recording unit that records the image data captured by each camera 21. For example, when a customer's disappearance or appearance behavior is recognized, the image data captured in the recording unit may be played back to verify the disappearance or appearance behavior.

[0125] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0126] 11...self-service POS terminal, 12...POS server, 13...display control device, 14...attendant terminal, 15...communication network, 21...camera, 22...information processing device, 40...main body, 41...touch panel, 42...reading window, 43...card insertion port, 44...receipt issuing port, 45...coin insertion port, 46...coin dispensing outlet, 47...banknote insertion port, 48...banknote dispensing outlet, 50...housing, 51...scale unit, 52...bag holder, 53...holding arm, 54...scale pan, 55...placing surface, 60...basket stand, 61...communication cable, 62...reader / writer, 63...stand, 64...display pole, 65...light emitting unit, 81...processor, 82...main memory, 83...auxiliary storage device, 84...clock, 85...timer, 86...camera interface, 87...communication interface, 88...system bus, 100...self-service POS system, 200...information processing system, 821...management table, 822...message table, 823...time series buffer, 221...behavior recognition unit, 222...operation recognition unit, 223...determination unit, 224...alarm unit.

Claims

1. an operation recognition means for recognizing an operation on the payment terminal by an operator; behavior recognition means for recognizing the behavior of the operator performing an operation on the payment terminal; a notification means for notifying the operator of a disappearance behavior of moving away from the payment terminal when the behavior recognition means recognizes the disappearance behavior of the operator moving away from the payment terminal before the operation recognition means recognizes the operation on the payment terminal as a payment completion operation; An information processing device comprising:

2. the notification means notifies the operator when an appearance behavior of the operator appearing at the payment terminal is recognized within a predetermined time after the disappearance behavior is recognized by the behavior recognition means; 2. The information processing device according to claim 1.

3. the notification means does not notify the fact that a cancel operation to cancel the operation on the payment terminal is recognized and the disappearance behavior is recognized by the behavior recognition means before the operation on the payment terminal is recognized by the operation recognition means as a payment completion operation; 2. The information processing device according to claim 1.

4. the operation recognition means recognizes an operation on the payment terminal by the operator based on an image that transitions in response to the operation on the payment terminal; 3. The information processing device according to claim 1.

5. an imaging device that captures an image of an operator operating the payment terminal; an information processing device including: an operation recognition means for recognizing an operation on the payment terminal by the operator; a behavior recognition means for recognizing an action of the operator based on image data of the operator captured by the imaging device; and a notification means for notifying the operator when the behavior recognition means recognizes a disappearance behavior of the operator moving away from the payment terminal before the operation recognition means recognizes the operation on the payment terminal as a payment completion operation; An information processing system comprising:

6. The computer of the information processing device, an operation recognition means for recognizing an operation on the payment terminal by an operator; A behavior recognition means for recognizing the behavior of the operator performing an operation on the payment terminal; a notification means for notifying the operator of a disappearance behavior of moving away from the payment terminal when the behavior recognition means recognizes the disappearance behavior of the operator moving away from the payment terminal before the operation recognition means recognizes the operation on the payment terminal as a payment completion operation; A control program that functions as a

Images