Information processing system, information processing device, and its control program

Abstract

To provide an information processing system capable of appropriately coping with a behavior in which a customer leaves a settlement terminal without completing the settlement, an information processing device and its control program.SOLUTION: An information processing device includes operation recognition means, behavior recognition means, and notification means. The operation recognition means recognizes operation to a settlement terminal by an operator. The behavior recognition means recognizes a behavior of the operator performing the operation to the settlement terminal. If the behavior recognition means recognizes a disappearance behavior in which the operator leaves the settlement terminal before the operation recognition means recognizes that the operation to the settlement terminal is settlement end operation, the notification means notifies the disappearance behavior.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing system, an information processing apparatus, and a control program thereof.

Background Art

[0002] In recent years, in retail stores such as supermarkets, self-POS (Point Of Sales) terminals have attracted attention from the viewpoints of reducing labor costs and preventing the spread of infectious diseases. A self-POS terminal is a full-self-service payment terminal that allows customers to perform operations from registering purchased goods to settlement by themselves. Therefore, there is a possibility that a situation may occur where a customer leaves the self-POS terminal without completing the settlement.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the embodiments of the present invention is to provide an information processing system, an information processing apparatus, and a control program thereof that can appropriately handle the behavior of a customer leaving the payment terminal without completing the settlement.

Means for Solving the Problems

[0005] In one embodiment, the information processing device comprises an operation recognition means, an action recognition means, an assignment means, an operation recognition means, a recording means, and a notification means. The action recognition means recognizes the appearance action of a person appearing at a payment terminal and the disappearance action of a person leaving the payment terminal. When the appearance action is recognized by the action recognition means, the assignment means assigns identification information to the person in question to identify them as an operator. The operation recognition means recognizes operations on the payment terminal. The recording means records the operations recognized by the operation recognition means as operations of an operator identified by the identification information assigned by the assignment means. If the disappearance action is recognized by the action recognition means and the payment completion operation as an operation on the payment terminal has not been recorded by the recording means, the notification means indicates the operator of the payment terminal. It disappeared during the payment process. A notification to prompt the attendant to confirm If the disappearance behavior is recognized and the reappearance behavior is recognized within a predetermined time after the disappearance behavior is recognized, the operator will notify the attendant to confirm the reappearance. conduct. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a system configuration diagram of a store where self-service POS terminals have been installed. [Figure 2] Figure 2 is a diagram illustrating the positional relationship between the self-service POS terminal and the camera. [Figure 3] Figure 3 is a schematic diagram showing an example of a monitoring image displayed on the attendant terminal's screen. [Figure 4] Figure 4 is a block diagram showing the main circuit configuration of the information processing device. [Figure 5] Figure 5 is a schematic diagram showing an example of a management table. [Figure 6] Figure 6 is a schematic diagram showing an example of a message table. [Figure 7] Figure 7 is a schematic diagram showing an example of a time-series buffer. [Figure 8] Figure 8 is a flowchart illustrating the function of the action recognition unit. [Figure 9] Figure 9 is a flowchart illustrating the function of the operation recognition unit. [Figure 10] Figure 10 is a flowchart illustrating the function of the operation recognition unit. [Figure 11] Figure 11 is a flowchart illustrating the function of the determination unit. [Figure 12] Figure 12 is a flowchart illustrating the function of the determination unit. [Figure 13] Figure 13 is a flowchart illustrating the functions of the notification unit. [Modes for carrying out the invention]

[0007] One embodiment will be described below with reference to the drawings. Figure 1 is a system configuration diagram of a store where self-service POS terminals 11 have been installed. This system includes a self-service POS system 100 and an information processing system 200. The self-service POS system 100 comprises multiple self-service POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The multiple self-service POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

[0008] The self-service POS terminal 11 is a fully self-service payment terminal that allows customers to perform operations from registering purchased items to payment themselves. The customer may also be referred to as the operator. The POS server 12 is a server computer for centrally controlling the operation of each self-service POS terminal 11. The display control device 13 is a controller that generates a monitoring image SC (see Figure 3) for each self-service POS terminal 11 based on the data signals output from each self-service POS terminal 11 and displays it on the display device of the attendant terminal 14. The attendant terminal 14 is a terminal used by store employees, referred to as attendants, to monitor the status of each self-service POS terminal 11. The attendant terminal 14 is equipped with a display such as a liquid crystal display or an organic EL (electro-luminescence) display. The attendant terminal 14 divides the display screen into multiple sections and displays a different monitoring image SC of the attendant terminal 14 in each section. Such a self-service POS system 100 can be implemented using conventionally known systems as they are.

[0009] The information processing system 200 includes multiple cameras 21 and an information processing device 22. Each of the multiple cameras 21 corresponds one-to-one with one of the multiple self-service POS terminals 11. The cameras 21 are for capturing images of customers operating the corresponding self-service POS terminals 11. The cameras 21 can also be referred to as imaging devices.

[0010] The information processing device 22 has the functions of an action recognition unit 221, an operation recognition unit 222, a determination unit 223, and a notification unit 224. The action recognition unit 221 has the function of recognizing the actions of a customer operating the self-service POS terminal 11 based on the image data output from each camera 21. The operation of the self-service POS terminal 11 will be clarified in the following description. The action recognition unit 221 can also be called an action recognition means. The operation recognition unit 222 has the function of recognizing the customer's operation of the self-service POS terminal 11 based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be called an operation recognition means. The determination unit 223 has the function of determining whether or not there is abnormal customer behavior based on the customer's actions toward the product for which the product registration operation has been recognized, based on the recognition results from the action recognition unit 221 and the recognition results from the operation recognition unit 222. The determination unit 223 can also be called a determination means. The notification unit 224 has the function of notifying the customer if the determination unit 223 determines whether or not the customer has engaged in abnormal behavior. The notification is made, for example, to the attendant from the attendant terminal 14. The notification unit 224 can also be called a notification means.

[0011] Figure 2 is a diagram illustrating the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external configuration of the self-service POS terminal 11 will be described.

[0012] The self-service POS terminal 11 comprises a main unit 40 installed on the floor and a weighing unit 51 installed next to the main unit 40. The main unit 40 has a touch panel 41 attached to its top. The touch panel 41 consists of a display and a touch sensor. The display is a device for showing various screens to the operator, i.e., the customer, who is operating the self-service POS terminal 11. The touch sensor is a device for detecting touch input on the screen by the customer.

[0013] The main body 40 is provided with a basket stand 60 at the center of the side surface on the side opposite to the side where the weighing unit 51 is installed. The basket stand 60 is for placing baskets or the like in which customers coming from the sales floor have put the purchased goods. The customer stands in front of the main body 40 in FIG. 2 to perform operations so that the screen of the touch panel 41 can be seen. Therefore, when viewed from the customer, there is a basket stand 60 on the right side across the main body 40, and a weighing unit 51 on the left side. In the following description, the side where the customer stands is taken as the front of the main body 40, the side where the weighing unit 51 is installed is taken as the left side of the main body 40, and the side where the basket stand 60 is provided is taken as the right side of the main body 40.

[0014] On the front of the main body 40, a reading window 42 of a scanner, a card insertion slot 43, a receipt issuing port 44, a coin insertion port 45, a coin payout port 46, a banknote insertion port 47, and a banknote payout port 48 are formed. Further, a communication cable 61 extends from the right side surface of the main body 40 to the outside, and a reader / writer 62 for an electronic money medium is connected to the tip of this communication cable 61. The reader / writer 62 is placed on a mounting base 63 provided on the upper part of the right side surface of the main body 40.

[0015] The main body 40 is attached with a display pole 64 on its upper surface. The display pole 64 is provided with a light emitting part 65 at its tip for selectively emitting, for example, blue and red light. The display pole 64 displays the state of the self-POS terminal 11, such as standby, operation, call, error, etc., according to the emission color of the light emitting part 65.

[0016] The weighing unit 51 has a weighing pan 54 provided on the upper part of the housing 50, and a bag holder 52 is attached above this weighing pan 54. The upper surface of the weighing pan 54 serves as a placement surface 55. The bag holder 52 includes a pair of holding arms 53, and the holding arms 53 hold a cash register bag or a shopping bag brought by the customer, so-called a my bag. The weighing unit 51 measures the weight of the goods placed on the placement surface 55 and put into a cash register bag or a my bag held by the holding arm 53.

[0017] Next, the positional relationship between the self-POS terminal 11 and the camera 21 will be described. As shown in Figure 2, the camera 21 is positioned in front of the self-service POS terminal 11, capable of capturing images from above of the customer facing the main unit 40, the weighing unit 51, and the basket stand 60.

[0018] A customer standing in front of the self-service POS terminal 11 first places a basket containing their purchased items on the basket tray 60 on the right, and then has the holding arm 53 on the left secure a shopping bag or reusable bag. Next, the customer follows the guidance displayed on the touch panel 41 and operates the touch panel 41 to declare that they are starting to use the self-service POS terminal 11.

[0019] Afterward, the customer picks up the items they wish to purchase one by one from the baskets placed on the basket stand 60. If the purchased items have barcodes, the customer registers the items by holding the barcodes up to the reading window 42 and having them scanned. If the purchased items do not have barcodes, the customer registers the items by operating the touch panel 41 and selecting them from the list of items without barcodes. The customer then places the registered purchased items into a shopping bag or their own reusable bag.

[0020] After registering all purchased items, the customer operates the touch panel 41 to select a payment method. For example, if cash payment is selected, the customer inserts banknotes or coins into the banknote slot 47 or coin slot 45 and takes the change dispensed from the banknote dispenser 48 or coin dispenser 46. For example, if electronic money payment is selected, the customer holds the electronic money medium over the reader / writer 62. For example, if credit card payment is selected, the customer inserts the credit card into the card slot 43. After completing the payment, the customer receives the receipt issued from the receipt dispenser 44 and leaves the store with the shopping bag or reusable bag removed from the holding arm 53.

[0021] Figure 3 is a schematic diagram showing an example of a monitoring image SC displayed on the attendant terminal 14's screen. The monitoring image SC is an example of an image that transitions in response to operations on the self-service POS terminal 11. As mentioned above, the attendant terminal 14's screen displays separate monitoring image SCs for each of the multiple self-service POS terminals 11. Figure 3 is an example of a monitoring image SC for one of these self-service POS terminals 11. The configuration of the monitoring image SCs for the other self-service POS terminals 11 is similar, so their explanation is omitted here.

[0022] As shown in Figure 3, the monitoring image SC includes a register number column 71, a terminal status column 72, an error information column 73, a declaration information column 74, a details column 75, and a total column 76.

[0023] The register number field 71 is a field for displaying the register number. The register number is a series of numbers assigned to each self-service POS terminal 11 in a non-repeating manner in order to individually identify each self-service POS terminal 11. The register number is identification information for identifying each self-service POS terminal 11.

[0024] The terminal status field 72 is a field for displaying the operating status of the self-service POS terminal 11. In this embodiment, one of the following operating statuses is displayed in the terminal status field 72: "Standby", "Starting Use", "Registering", "Starting Payment", and "Paying".

[0025] "Waiting" is the state between the time the previous customer has finished payment and the declaration that the next customer has started using the service. When the self-service POS terminal 11 is in the "Waiting" state, an initial image is displayed on the touch panel 41. The initial image is, for example, an image that includes touch buttons to allow the customer to select whether to use a store-provided shopping bag or a reusable bag. Also, if the operating status of the self-service POS terminal 11 is "Starting Use," "Registering," or "Starting Payment," and no operation is recognized on the self-service POS terminal 11 for a set period of time, the operating status of the self-service POS terminal 11 will return to "Waiting." The set period can be, for example, 30 seconds or 1 minute. The set period can be set arbitrarily.

[0026] "Start of Use" is the state in which a customer standing in front of the self-service POS terminal 11 declares that they want to start using the service for payment. The customer makes a selection operation on the initial image, indicating whether to use a plastic shopping bag or a reusable bag. This selection operation constitutes the declaration of starting use. Upon receiving this selection operation, the operating state of the self-service POS terminal 11 becomes "Start of Use".

[0027] The "Registering" state indicates that the terminal is accepting the customer's attempt to register their purchased items. Once the first purchased item is registered, the self-POS terminal 11 will enter the "Registering" state. The self-POS terminal 11 will remain in the "Registering" state until the customer declares they are ready to proceed to payment. Additionally, the touch panel 41 of the self-POS terminal 11 in the "Registering" state will display a soft key, such as a [Cancel Registration] key. A customer who wishes to cancel the registration of their purchased items will touch the [Cancel Registration] soft key. Touching the [Cancel Registration] soft key is an example of a cancellation operation to stop interacting with the self-POS terminal 11. Upon receiving this cancellation operation, the self-POS terminal 11 will return to the "Standby" state.

[0028] "Payment Started" is the state in which a customer who has finished registering their purchased items declares that they wish to proceed to payment. The touch panel 41 of the self-service POS terminal 11, which is in the "Registering" state, displays the [Checkout] soft key. The customer who has finished registering their purchased items touches the [Checkout] soft key. This action constitutes a declaration to proceed to payment. Upon receiving this action, the operating state of the self-service POS terminal 11 changes to "Payment Started". After "Payment Started", the touch panel 41 of the self-service POS terminal 11 displays a soft key, for example, [Cancel Payment]. The customer who wishes to cancel the payment for their purchased items touches the [Cancel Payment] soft key. The customer touching the [Cancel Payment] soft key is an example of a cancellation operation. Upon receiving this cancellation operation, the operating state of the self-service POS terminal 11 returns to "Standby".

[0029] "Payment in progress" indicates a state in which payment processing is being performed, such as cash payment, electronic money payment, or credit card payment. For example, when banknotes or coins are inserted into the banknote slot 47 or coin slot 45, the operating status of the self-POS terminal 11 becomes "Payment in progress." After the payment processing is complete, the operating status of the self-POS terminal 11 returns to "Standby."

[0030] Error information field 73 is for displaying error information that occurred at the self-service POS terminal 11. Error information includes, for example, communication errors and receipt out errors. Declaration information field 74 is for displaying the customer's declared actions. For example, if the customer selects their own bag, "No bag needed" will be displayed to indicate that a plastic bag is not required.

[0031] The details column 75 is for displaying the details of the purchased items registered at the self-service POS terminal 11. The details include, for example, the name of the purchased item, the quantity, and the price. The total column 76 is for displaying the total information of the purchased items registered at the self-service POS terminal 11. The total information includes, for example, the total number of items, the total amount, the amount paid, and the change received.

[0032] The configuration of the monitoring image SC is not limited to this. Fields for displaying other items may be included. Furthermore, the items of text data shown in Figure 3 are not limited to these. Text data for other items may also be displayed.

[0033] Figure 4 is a block diagram showing the main circuit configuration of the information processing device 22. The information processing device 22 comprises a processor 81, main memory 82, auxiliary storage device 83, clock 84, timer 85, camera interface 86, communication interface 87, and system bus 88. The system bus 88 includes an address bus, a data bus, etc. The information processing device 22 constitutes a computer by connecting the processor 81, the main memory 82, auxiliary storage device 83, clock 84, timer 85, camera interface 86, and communication interface 87 via the system bus 88.

[0034] The processor 81 corresponds to the central part of the computer described above. The processor 81 controls each part in order to realize various functions as an information processing device 22 according to the operating system or application program. The processor 81 is, for example, a CPU (Central Processing Unit).

[0035] Main memory 82 corresponds to the main memory portion of the computer described above. Main memory 82 includes non-volatile memory areas and volatile memory areas. In the non-volatile memory area of ​​main memory 82, the operating system or application programs are stored. In the volatile memory area of ​​main memory 82, data necessary for the processor 81 to perform processing to control each part is stored. This type of data may also be stored in the non-volatile memory area. Main memory 82 uses the volatile memory area as a work area where data is rewritten as needed by the processor 81. The non-volatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0036] The auxiliary storage device 83 corresponds to the auxiliary storage portion of the computer described above. The auxiliary storage device 83 may be a well-known storage device such as an SSD (Solid State Drive), HDD (Hard Disk Drive), or EEPROM (Registered Trademark) (Electric Erasable Programmable Read-Only Memory), used alone or in combination with others. The auxiliary storage device 83 stores data used by the processor 81 for various processing tasks, data generated by the processing performed by the processor 81, and so on. The auxiliary storage device 83 may also store application programs.

[0037] The application program stored in the main memory 82 or auxiliary storage device 83 includes the control program described later. The method of installing the control program in the main memory 82 or auxiliary storage device 83 is not particularly limited. The control program can be recorded on a removable recording medium or distributed via network communication and installed in the main memory 82 or auxiliary storage device 83. The recording medium can be of any form as long as it can store a program and is readable by the device, such as a CD-ROM or memory card.

[0038] The clock 84 functions as a time information source for the information processing device 22. The processor 81 obtains the current date and time based on the time information measured by the clock 84.

[0039] Timer 85 measures the set time under the control of processor 81.

[0040] The camera interface 86 is an interface for communicating with each camera 21. The image data output from each camera 21 is received by the information processing device 22 via the camera interface 86. The image data is data of customers operating the self-service POS terminal 11 corresponding to the camera 21.

[0041] The communication interface 87 is an interface for communicating with the display control device 13. Image data output from the display control device 13 is received by the information processing device 22 via the communication interface 87. The image data is the data of the monitoring image SC generated for each self-service POS terminal 11.

[0042] In this configuration, the information processing device 22 allocates a portion of the volatile memory area in the main memory 82 to the management table 821 (see Figure 5) and the message table 822 (see Figure 6). The information processing device 22 then forms the management table 821 and the message table 822 in this area.

[0043] Figure 5 is a schematic diagram showing an example of a management table 821. As shown in Figure 5, the management table 821 has an area for describing one or more tracking IDs associated with the register number of the self-service POS terminal 11. The tracking ID is identification information used to temporarily identify a person as a tracking target when it is detected that a person is standing in front of the self-service POS terminal 11. Note that the management table 821 is not limited to the data of the items described above.

[0044] Figure 6 is a schematic diagram showing an example of a message table 822. As shown in Figure 6, the message table 822 is a data table that describes the text data of a message in relation to the message code MC. The message code MC and the text data of the message associated with it will be explained later.

[0045] Furthermore, the information processing device 22 allocates a portion of the volatile memory area in the main memory 82 to the time-series buffer 823 (see Figure 7). The information processing device 22 then creates as many time-series buffers 823 as there are self-service POS terminals 11 in this area.

[0046] Figure 7 is a schematic diagram showing an example of a time-series buffer 823. As shown in Figure 7, the time-series buffer 823 has areas for describing the tracking ID, time TM, and status ST or message code MC for each register number that identifies the self-service POS terminal 11. In the time-series buffer 823, the status ST or message code MC is described in order of earliest time TM. The status ST will be explained in a later section.

[0047] However, the information processing device 22 implements the functions of the aforementioned action recognition unit 221, operation recognition unit 222, determination unit 223, and notification unit 224 through a processor 81 and a control program that controls the processor 81.

[0048] The functions of the action recognition unit 221, operation recognition unit 222, judgment unit 223, and notification unit 224 are provided for each self-service POS terminal 11. Therefore, the functions of the action recognition unit 221, operation recognition unit 222, judgment unit 223, and notification unit 224 for one self-service POS terminal 11 will be described in detail below. The functions of the action recognition unit 221, operation recognition unit 222, judgment unit 223, and notification unit 224 for other self-service POS terminals 11 are the same, so the explanation will be omitted here.

[0049] Figure 8 is a flowchart illustrating the function of the action recognition unit 221. The processor 81 waits to recognize a customer's appearance as ACT1. An appearance is the action of a customer appearing at the self-service POS terminal 11. When the processor 81 detects from the image data of the camera 21 that a person is standing in front of the self-service POS terminal 11, it recognizes that an appearance has occurred. Examples of cases in which a customer is detected from the image data of the camera 21 include when a customer performs operations from registering purchased items to payment at the self-service POS terminal 11, or when a customer returns to the self-service POS terminal 11 after going to the sales floor to pick up an item they forgot to buy.

[0050] Upon recognizing the customer's appearance, processor 81 determines YES in ACT1 and proceeds to ACT2. In ACT2, processor 81 assigns a tracking ID to that customer.

[0051] The processor 81 obtains the register number of the self-service POS terminal 11 as ACT3. Each camera 21 corresponds one-to-one with each self-service POS terminal 11. The processor 81 then identifies the self-service POS terminal 11 from the identification information of the camera 21 that captures images of the customer standing in front of the self-service POS terminal 11, and obtains the register number of that self-service POS terminal 11.

[0052] The processor 81, as ACT4, writes the tracking ID assigned in the ACT2 process to the management table 821 of the main memory 82, associating it with the register number of the self-service POS terminal 11 obtained in the ACT3 process.

[0053] Processor 81 sets the status ST to "11" as ACT5. That is, if the action of the self-POS terminal 11 is recognized by the action recognition unit 221, the status ST is set to "11". Processor 81 obtains the current time TM, which is measured by the clock 84, as ACT6. Processor 81, as ACT7, writes the tracking ID, time TM, and status ST assigned in the ACT2 process to the time-series buffer 823, which is set with the register number obtained in the ACT3 process, associating them.

[0054] The processor 81 waits to recognize a customer's disappearance as ACT8. Disappearance is the action of a customer leaving the self-POS terminal 11. The processor 81 recognizes that a disappearance has occurred when it can no longer detect the customer from the image data captured by the camera 21. Situations in which the customer can no longer be detected from the image data captured by the camera 21 include, for example, when a customer leaves the self-POS terminal 11 with purchased items that have not yet been paid for, when a customer leaves the self-POS terminal 11 to leave the store after completing payment, when a customer leaves the self-POS terminal 11 to go to the sales floor to retrieve items they forgot to buy, or when a customer leaves the self-POS terminal 11 after performing a cancellation operation.

[0055] Upon recognizing the disappearance behavior, the processor 81 determines YES in ACT8 and proceeds to ACT9. In ACT9, the processor 81 sets the status ST to "12". That is, when the behavior recognition unit 221 recognizes the disappearance behavior toward the self-POS terminal 11, the status ST is set to "12".

[0056] Processor 81 obtains the current time TM, which is measured by clock 84, as ACT10. Processor 81 then writes the tracking ID, time TM, and status ST, which were assigned in ACT2, to the time-series buffer 823, which is set with the registration number obtained in ACT3, as ACT11. Processor 81 then returns to ACT1. That is, when it is detected again from the image data of camera 21 that a person is standing in front of the self-POS terminal 11, processor 81 executes the processes of ACT2 to ACT11 in the same manner as described above. This concludes the explanation of the function of the action recognition unit 221.

[0057] Figure 9 is a flowchart illustrating the function of the operation recognition unit 222. The processor 81 waits for a declaration of activation for the self-service POS terminal 11 as ACT21. When activation is declared, "Activation Started" is displayed in the terminal status column 72 of the monitoring image SC corresponding to the self-service POS terminal 11. The processor 81 checks whether it can recognize the characters "Activation Started" from the terminal status column 72 of the monitoring image SC acquired via the display control device 13. If the characters "Activation Started" are recognized, the processor 81 recognizes that activation has been declared.

[0058] Upon recognizing that the start of use has been declared, the processor 81 determines YES in ACT21 and proceeds to ACT22. In ACT22, the processor 81 obtains the register number of the self-service POS terminal 11. The register number is displayed in the register number field 71 of the monitoring image SC. The processor 81 recognizes the characters of the register number from the register number field 71 of the monitoring image SC obtained via the display control device 13 and obtains those characters as the register number.

[0059] Processor 81 sets the status ST to "21" as ACT23. Processor 81 obtains the current time TM, which is being measured by clock 84, as ACT24.

[0060] Processor 81 obtains a tracking ID by referring to the management table 821 as ACT25. Specifically, processor 81 refers to the management table 821 and obtains the latest tracking ID that is associated with the register number obtained in ACT22. The latest tracking ID is the tracking ID if there is one tracking ID associated with the register number. The latest tracking ID is the next tracking ID if there are multiple tracking IDs associated with the register number. The specific processing of ACT38, ACT42, and ACT47 in Figure 10, which will be described later, is the same as the processing of ACT25.

[0061] The processor 81, as ACT26, writes the tracking ID, time TM, and status ST obtained in the processing of ACT25 to the time-series buffer 823, which is set to the registration number obtained in the processing of ACT22, associating them.

[0062] Therefore, when a customer standing in front of the self-service POS terminal 11 performs the declaration operation to start using the service, the time-series buffer 823 corresponding to the self-service POS terminal 11 first records "21" as the status ST, along with its time TM and tracking ID.

[0063] The processor 81 initiates operation recognition for the self-service POS terminal 11 as ACT27. Specifically, the processor 81 recognizes product registration operation, payment start operation, cancellation operation, or payment end operation from the information transitions obtained by character recognition of the monitoring image SC acquired via the display control device 13.

[0064] For example, processor 81 recognizes that a product registration operation has occurred when detailed information such as the product name, quantity, and price of the purchased product is added to the details field 75. For example, processor 81 recognizes that a payment start operation has occurred when the display in the terminal status field 72 switches to "Payment Started". For example, processor 81 recognizes that a cancellation operation has occurred when the display in the terminal status field 72 switches from "Registering" to "Waiting". For example, processor 81 recognizes that a cancellation operation has occurred when the display in the terminal status field 72 switches from "Payment In Progress" to "Waiting". For example, processor 81 recognizes that a payment completion operation has occurred when the display in the terminal status field 72 switches from "Payment in Progress" to "Waiting".

[0065] The processor 81 waits to recognize product registration operations, payment start operations, cancellation operations, or payment end operations as ACT31 to ACT34 in Figure 10.

[0066] In the waiting state of ACT31 to ACT34, when the processor 81 recognizes a product registration operation, it determines YES in ACT31 and proceeds to ACT35. In ACT35, the processor 81 sets the status ST to "22". That is, when a product registration operation to the self-POS terminal 11 is recognized by the operation recognition unit 222, the status ST is set to "22".

[0067] In the waiting state of ACT31 to ACT34, when the processor 81 recognizes a payment initiation operation, it determines YES in ACT32 and proceeds to ACT36. In ACT36, the processor 81 sets the status ST to "23". That is, when the operation recognition unit 222 recognizes a payment initiation operation for the self-POS terminal 11, the status ST is set to "23".

[0068] After completing the processing of ACT35 or ACT36, processor 81 proceeds to ACT37. As ACT37, processor 81 obtains the current time TM, which is measured by clock 84.

[0069] Processor 81 obtains the tracking ID by referring to the management table 821 as ACT38. Processor 81 then writes the tracking ID obtained in the ACT38 process, along with the time TM and status ST, to the time-series buffer 823, which is set to the registration number obtained in the ACT22 process in Figure 9, as ACT39. Processor 81 then returns to the waiting state of ACT31 to ACT34.

[0070] In the waiting state of ACT31 to ACT34, when the processor 81 recognizes a cancellation operation, it determines YES in ACT33 and proceeds to ACT40. In ACT40, the processor 81 sets the status ST to "24". That is, when a cancellation operation to the self-POS terminal 11 is recognized by the operation recognition unit 222, the status ST is set to "24".

[0071] Processor 81 obtains the current time TM, which is measured by clock 84, as ACT41. Processor 81 obtains the tracking ID by referring to the management table 821 as ACT42. Processor 81, as ACT43, writes the tracking ID, time TM, and status ST obtained in the process of ACT42 to the time-series buffer 823, which is set with the registration number obtained in the process of ACT22 in Figure 9, associating them. With this, processor 81 terminates its function as the operation recognition unit 222.

[0072] In the waiting state of ACT31 through ACT34, if the processor 81 recognizes a payment completion operation, it determines YES in ACT34 and proceeds to ACT44. The processor 81 then terminates the recognition of operations for the self-service POS terminal 11 in ACT44.

[0073] The processor 81, having completed operation recognition, sets its status ST to "25" as ACT45. In other words, when the operation recognition unit 222 recognizes that a payment completion operation has been completed for the self-service POS terminal 11, the status ST is set to "25".

[0074] Processor 81 obtains the current time TM, which is measured by clock 84, as ACT46. Processor 81 obtains the tracking ID by referring to the management table 821 as ACT47. Processor 81, as ACT48, writes the tracking ID, time TM, and status ST obtained in the process of ACT47 to the time-series buffer 823, which is set with the registration number obtained in the process of ACT22 in Figure 9, associating them. With this, processor 81 terminates its function as the operation recognition unit 222.

[0075] Typically, customers perform the following operations on the self-service POS terminal 11: start operation, product registration operation, payment start operation, and payment end operation. Therefore, the status ST will be recorded in the time-series buffer 823 in the order of "21", "22", "23", and "25".

[0076] A customer who wishes to cancel the registration of purchased items performs the following operations on the self-service POS terminal 11 in the order of activation, product registration, and cancellation. Therefore, the status ST will be recorded in the time-series buffer 823 in the order of "21", "22", and "24".

[0077] A customer who wishes to cancel the payment for their purchased items performs the following operations on the self-service POS terminal 11 in the order of activation, product registration, payment activation, and cancellation. Therefore, the status ST will be recorded in the time-series buffer 823 in the order of "21", "22", "23", and "24".

[0078] This concludes the explanation of the operation recognition unit 222's function. Subsequently, when the data of the monitoring image SC detects the start operation of the self-POS terminal 11, the processor 81 executes the processes of ACT22 to ACT27 in Figure 9 and ACT31 to ACT48 in Figure 10 in the same manner as described above.

[0079] Figures 11 and 12 are flowcharts illustrating the function of the determination unit 223. The processor 81 checks whether the behavior has been recognized by the behavior recognition unit 221 as ACT51. If the behavior is recognized, "11" is written as status ST in the time series buffer 823. If "11" is written as status ST in the time series buffer 823, the processor 81 determines YES in ACT51 and proceeds to ACT52.

[0080] The processor 81 checks in ACT52 whether the operation to start using the service has been recognized by the operation recognition unit 222. When the operation to start using the service is performed on the self-service POS terminal 11, "21" is written as the status ST in the time-series buffer 823, which is set to the register number of the self-service POS terminal 11 on which the operation to start using the service was performed. If "21" is written as the status ST in the time-series buffer 823, the processor 81 determines YES in ACT52 and proceeds to ACT53.

[0081] The processor 81 checks whether the disappearance behavior has been recognized by the behavior recognition unit 221 as ACT53. If the disappearance behavior is recognized, "12" is written as the status ST in the time series buffer 823. Therefore, if "12" is written as the status ST in the time series buffer 823, the processor 81 determines YES in ACT53 and proceeds to ACT54.

[0082] Processor 81 checks, as ACT54, whether the disappearance behavior occurred after a cancellation operation. When a cancellation operation is performed on the self-POS terminal 11, "24" is written as status ST in the time series buffer 823. Therefore, if "24" is written as status ST in the time series buffer 823 in association with a time TM one time earlier than the time TM in which "12" was written as status ST, processor 81 determines that the disappearance behavior occurred after a cancellation operation.

[0083] If the disappearance action does not occur after a cancellation operation, the processor 81 determines NO in ACT 54 and proceeds to ACT 55. In ACT 55, the processor 81 checks whether the disappearance action occurred after a product registration operation. When a product registration operation is performed on the self-POS terminal 11, "22" is written as the status ST in the time series buffer 823. Therefore, if "22" is written as the status ST in the time series buffer 823 in association with a time TM one time earlier than the time TM in which "12" was written as the status ST, the processor 81 determines that the disappearance action occurred after a product registration operation.

[0084] If the disappearance action does not occur after the product registration operation, the processor 81 determines NO in ACT 55 and proceeds to ACT 56. In ACT 56, the processor 81 checks whether the disappearance action occurred after the payment initiation operation. When the payment initiation operation is performed on the self-POS terminal 11, "23" is written as the status ST in the time series buffer 823. Therefore, if "23" is written as the status ST in the time series buffer 823 in association with a time TM one time earlier than the time TM in which "12" is written as the status ST, the processor 81 determines that the disappearance action occurred after the payment initiation operation.

[0085] If the disappearance behavior does not occur after the payment start operation, i.e., if the disappearance behavior occurs after the payment end operation, the processor 81 determines NO in ACT 56. If the customer's disappearance behavior is recognized at the self-POS terminal 11 where the payment end operation was performed, for example, a customer who has finished payment may leave the self-POS terminal 11 in order to leave the store. This customer's action is determined not to be an abnormal act. At this point, the processor 81 terminates its function as the determination unit 223.

[0086] In the standby state of ACT54 to ACT56, if the disappearance occurs after a cancellation operation, the processor 81 determines YES in ACT54. If the customer's disappearance is recognized at the self-POS terminal 11 where the cancellation operation was performed, for example, the customer may have touched the [Cancel Registration] soft key to cancel the registration of purchased items or the [Cancel Payment] soft key to cancel the payment of purchased items displayed on the touch panel 41 and then left the self-POS terminal 11. This customer's action is determined not to be an abnormal act. At this point, the processor 81 terminates its function as the determination unit 223.

[0087] In the waiting state of ACT54 through ACT56, if the disappearance action occurs after the product registration operation, the processor 81 determines YES in ACT55 and proceeds to ACT57. The processor 81 sets the message code MC to "91" in ACT57.

[0088] If a customer disappears from a self-service POS terminal 11 where a product registration operation has been performed, for example, if the customer leaves the self-service POS terminal 11 with purchased items that have not yet been paid for, this customer's behavior is determined to be abnormal. The message code MC "91" is written to the time-series buffer 823, which is set to the register number of the self-service POS terminal 11. The message code MC "91" is a code that identifies "disappearance of a customer during a product registration operation."

[0089] In the waiting state of ACT54 through ACT56, if the disappearance action occurs after the settlement start operation, the processor 81 determines YES in ACT56 and proceeds to ACT58. The processor 81 sets the message code MC to "92" for ACT58.

[0090] If a customer disappears from a self-service POS terminal 11 where a payment initiation operation has been performed, for example, if the customer leaves the self-service POS terminal 11 with purchased items that have not yet been paid for, this customer's behavior is determined to be abnormal. The message code MC "92" is written to the time-series buffer 823, which is set to the register number of the self-service POS terminal 11. The message code MC "92" is a code that identifies "disappearance of a customer during a payment operation."

[0091] After completing the processing of ACT57 or ACT58, processor 81 proceeds to ACT59. As ACT59, processor 81 obtains the current time TM, which is being measured by clock 84.

[0092] Processor 81 obtains a tracking ID by referring to the management table 821 as ACT60. Specifically, processor 81 refers to the management table 821 and obtains the latest tracking ID that is associated with the register number of the self-service POS terminal 11 for which the activation operation was recognized. The latest tracking ID is the tracking ID if there is one tracking ID associated with the register number. The latest tracking ID is the next tracking ID if there are multiple tracking IDs associated with the register number. The specific processing of ACT76 in Figure 12, which will be described later, is the same as the processing of ACT60.

[0093] The processor 81 writes the tracking ID, time TM, and message code MC obtained in the processing of ACT60 to the time series buffer 823 as ACT61, associating them with each other.

[0094] The processor 81 checks whether the appearance behavior has been recognized by the behavior recognition unit 221 as ACT71 in Figure 12. If the appearance behavior is recognized, "11" is written as the status ST in the time series buffer 823. Therefore, if "12" is written as the status ST in the time series buffer 823 in association with a time TM one time before the time TM in which "11" is written as the status ST, the processor 81 determines that it is an appearance behavior after the disappearance behavior.

[0095] Processor 81 calculates the time difference between the time TM in which status ST "11" is described and the time TM immediately preceding that time TM, in which status ST "12" is described. In other words, Processor 81 calculates the time difference from the time TM in which the disappearance behavior is recognized by the behavior recognition unit 221 to the time TM in which the appearance behavior is recognized.

[0096] The processor 81 checks, as ACT73, whether the time difference is within a predetermined time. The predetermined time is, for example, 30 seconds, 1 minute, etc. The predetermined time can be set arbitrarily.

[0097] If the time difference is not within the predetermined time, i.e., if the time difference is outside the predetermined time, the processor 81 determines NO in ACT73 and terminates its function as the determination unit 223.

[0098] If the time difference is within the predetermined time, the processor 81 determines YES in ACT73 and proceeds to ACT74. The processor 81 sets the message code MC to "93" in ACT74.

[0099] In a self-service POS terminal 11 where a customer's disappearance has been recognized, if the customer's reappearance is recognized within a predetermined time, for example, if a customer leaves the self-service POS terminal 11 to retrieve an item they forgot to buy from the sales floor and then returns to the self-service POS terminal 11, it is conceivable that a different customer than the one who left the self-service POS terminal 11 with the unpaid purchased items will perform the operations from registering the purchased items to completing the payment at the self-service POS terminal 11. The message code MC "93" is a code that identifies "customer reappearance".

[0100] Processor 81 obtains the current time TM, which is measured by clock 84, as ACT75. Processor 81 obtains the tracking ID by referring to management table 821 as ACT76. Processor 81 writes the tracking ID, time TM, and message code MC obtained in the processing of ACT76 to the time series buffer 823 as ACT77. Then processor 81 returns to ACT53 in Figure 11.

[0101] This concludes the explanation of the function of the determination unit 223. Subsequently, when an occurrence of behavior is recognized again, the processor 81 executes the processes of ACT52 to ACT61 in Figure 11 and ACT71 to ACT77 in Figure 12 in the same manner as described above.

[0102] Figure 13 is a flowchart illustrating the functions of the notification unit 224. Processor 81 waits for the message code MC to be written to the time-series buffer 823 as ACT81. When the message code MC is written to the time-series buffer 823, processor 81 determines YES in ACT81 and proceeds to ACT82. Processor 81 checks whether the message code MC is "91" as ACT82.

[0103] If the message code MC is "91", the processor 81 determines YES in ACT 82 and proceeds to ACT 84. In ACT 84, the processor 81 notifies that "the customer has disappeared during product registration." Specifically, the processor 81 searches the message table 822 and retrieves the message data with message code MC "91". The processor 81 also retrieves the register number in the time-series buffer 823 where message code MC "91" is written. The processor 81 then outputs a notification command with the message data and register number attached to it from the communication interface 87 to the display control device 13.

[0104] The display control device 13 causes the text of the message data to be displayed on the monitoring image SC of the attendant terminal 14, which is identified by the register number included in the notification command. As a result, the monitoring image SC displays text such as, "A customer disappeared while registering an item at register No. X."

[0105] If the message code MC is not "91", processor 81 determines NO in ACT82 and proceeds to ACT83. Processor 81 then checks in ACT83 whether the message code MC is "92".

[0106] If the message code MC is "92", the processor 81 determines YES in ACT 83 and proceeds to ACT 84. In ACT 84, the processor 81 notifies that "a customer has disappeared during a payment transaction". Specifically, the processor 81 searches the message table 822 and retrieves the message data with message code MC "92". The processor 81 also retrieves the register number in the time-series buffer 823 where "92" is written as the message code MC. The processor 81 then outputs a notification command with the message data and register number attached to it from the communication interface 87 to the display control device 13.

[0107] The display control device 13 causes the text of the message data to be displayed on the monitoring image SC of the attendant terminal 14, which is identified by the register number included in the notification command. As a result, the monitoring image SC displays text such as, "A customer disappeared while a payment operation was being performed at register No. X."

[0108] Thus, the notification unit 224, when the message code MC is "91" or "92", notifies that an abnormal act has occurred, specifically "disappearance of a customer during product registration" or "disappearance of a customer during payment." The attendant then takes action, for example, by rushing to the self-POS terminal 11 set to register no. X to check the situation or by searching for the customer who committed the abnormal act.

[0109] If the message code MC is not "92", that is, if the message code MC is "93", the processor 81 determines NO in ACT 83 and proceeds to ACT 85. The processor 81 then notifies "customer appearance" as ACT 85. Specifically, the processor 81 searches the message table 822 and retrieves the message data with message code MC "93". The processor 81 also retrieves the register number of the time-series buffer 823 where "93" is written as the message code MC. The processor 81 then outputs a notification command with the message data and register number attached to it from the communication interface 87 to the display control device 13.

[0110] The display control device 13 causes the text of the message data to be displayed on the monitoring image SC of the attendant terminal 14, which is identified by the register number included in the notification command. As a result, the monitoring image SC displays text such as "A customer has appeared at register No. X."

[0111] Thus, due to the function of the notification unit 224, if "93" is written as the message code MC in the time-series buffer 823, a notification of "customer appearance" is sent. The attendant then takes action, for example, by confirming that the customer has appeared at the self-POS terminal 11 set to register No. X, and checking whether the customer has gone to the sales floor to retrieve forgotten items and returned to the self-POS terminal 11.

[0112] After completing the processing of ACT84 or ACT85, the processor 81 returns to ACT81. That is, when the message code MC is written to the time-series buffer 823, the processor 81 executes the processing of ACT82 to ACT85 in the same manner as described above. This concludes the explanation of the functions of the notification unit 224.

[0113] As detailed above, the information processing device 22, as an action recognition unit 221, recognizes the actions of a customer operating the self-service POS terminal 11 based on the image data captured by the camera 21. The information processing device 22, as an operation recognition unit 222, recognizes the customer's operations on the self-service POS terminal 11. The customer's operations on the self-service POS terminal 11 are recognized based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14. The information processing device 22, as a determination unit 223, determines whether or not there is abnormal behavior by the customer based on the customer's actions toward the product for which the product registration operation has been recognized, based on the recognition results from the action recognition unit 221 and the recognition results from the operation recognition unit 222. The information processing device 22, as a notification unit 224, notifies the customer of abnormal behavior if the determination unit 223 has determined whether or not there is abnormal behavior.

[0114] For example, if a customer's disappearance is detected at a self-service POS terminal 11 where a cancellation or payment completion operation has been performed, the customer's actions are determined not to constitute abnormal behavior. In this case, the attendant terminal 14 is not used to notify the attendant of the customer's disappearance. On the other hand, if a customer disappears from a self-service POS terminal 11 where, for example, a product registration operation or payment initiation operation is performed without a cancellation operation being performed, the customer's actions are determined to be abnormal. In this case, the attendant is notified via the attendant terminal 14 that "a customer has disappeared during a product registration operation" or "a customer has disappeared during a payment operation." Therefore, attendants can take action such as rushing to the self-service POS terminal 11 to check the situation or searching for the customer who engaged in abnormal behavior.

[0115] For example, if the customer reappears at the self-service POS terminal 11 where the customer's disappearance was recognized within a predetermined time after the attendant is notified of "the customer disappearing while registering products" or "the customer disappearing while making a payment," the attendant terminal 14 is used to notify the attendant of "the customer's reappearance." Therefore, the attendant can take actions such as confirming that the customer has reappeared at the self-service POS terminal 11, and checking whether the customer has gone to the sales floor to retrieve forgotten items and returned to the self-service POS terminal 11.

[0116] Thus, according to this embodiment, it is possible to appropriately deal with the act of a customer leaving the self-service POS terminal 11 before completing payment.

[0117] The embodiments of the information processing system, information processing device, and control program thereof have been described above, but the embodiments are not limited thereto.

[0118] In the above embodiment, an example was given in which one camera 21 is placed for one self-service POS terminal 11. A camera 21 does not necessarily have to be placed for each self-service POS terminal 11. For example, if a single camera 21 can capture images of customers operating two adjacent self-service POS terminals 11, the number of cameras 21 may be reduced. However, in that case, in ACT3 of Figure 8, the register number of the self-service POS terminal 11 closest to the position of the person captured in the image data will be obtained. Alternatively, multiple cameras 21 may be placed for a single self-service POS terminal 11. By doing so, blind spots of the self-service POS terminal 11 can be reduced, and customer behavior can be detected with higher accuracy.

[0119] In the above embodiment, when a customer disappears from a self-service POS terminal 11 where a product registration operation or payment initiation operation has been performed without a cancellation operation, the attendant terminal 14 is used to notify the attendant of "the customer has disappeared during the product registration operation" or "the customer has disappeared during the payment operation." For example, if a customer who has performed a product registration operation or payment initiation operation without canceling the operation leaves the self-POS terminal 11 with the purchased items in their hands, the attendant may be notified that "a customer has disappeared during product registration" or "a customer has disappeared during payment." In other words, if a customer leaves the self-POS terminal 11 without the purchased items in their hands, the attendant does not need to be notified that "a customer has disappeared during product registration" or "a customer has disappeared during payment." In this case, the processor 81 estimates the person, i.e., the customer's skeleton, from the image data captured by the camera 21. Skeleton estimation can be achieved even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the movement of the customer picking up the purchased items from the hand movements obtained by skeletal estimation. For example, if a customer who has performed a product registration operation or payment initiation operation without canceling the operation leaves the self-POS terminal 11, and the total weight of the purchased items measured by the weighing unit 51 is detected to have changed in a decreasing direction, the attendant may be notified of "disappearance of customer during product registration operation" or "disappearance of customer during payment operation." In other words, if the total weight of the purchased items measured by the weighing unit 51 does not change and the customer leaves the self-POS terminal 11, it is not necessary to notify the attendant of "disappearance of customer during product registration operation" or "disappearance of customer during payment operation."

[0120] In the above embodiment, it was illustrated that when the processor 81 recognizes the customer's appearance, it assigns a tracking ID to that customer. For example, the processor 81 may identify the customer based on the clothes the customer is wearing from the image data captured by the camera 21. For example, if a customer leaves the self-POS terminal 11 to retrieve an item they forgot to buy and then returns to the self-POS terminal 11, the customer who left the self-POS terminal 11 and the customer who returned to the self-POS terminal 11 are the same person, and therefore their clothes are the same. In this case, even if the processor 81 recognizes the customer's appearance, it does not assign a new tracking ID because it considers them to be the same person. For example, if a customer leaves the self-POS terminal 11 with purchased items that have not yet been paid for, and then a different customer performs the operations from registering the purchased items to paying for them at the self-POS terminal 11, the clothes will be different. In this case, the processor 81 assigns a tracking ID to that customer because it considers them to be a different person.

[0121] In the above embodiment, the notification unit 224 is shown as an example of notifying an attendant via the attendant terminal 14. The notification destination is not limited to the attendant terminal 14. For example, the self-POS terminal 11 where the abnormal behavior occurred may be the notification destination. In this case, for example, the light-emitting unit 65 may emit light in a predetermined color to inform the store staff that the disappearance behavior is occurring. Alternatively, the touch panel 41 may display a message or provide an audio message to inform the staff that the disappearance behavior is occurring.

[0122] In the above embodiment, the attendant terminal 14 may also function as a display control device 13. In this case, the operation recognition unit 222 acquires data of the monitoring image SC from the attendant terminal 14 and recognizes the customer's operation on the self-POS terminal 11. Alternatively, the operation recognition unit 222 may acquire data signals output from each self-POS terminal 11 from the communication network 15, for example via a router, and recognize the customer's operation on the self-POS terminal 11 based on these data signals.

[0123] In the above embodiment, the case of a self-service POS terminal 11 was illustrated. For example, a semi-self-service POS terminal may be used in which the store clerk registers the purchased items and the customer makes the payment. In this case, the message table 822 includes message codes MC "92" and "93". That is, the message table 822 does not include message code MC "91", which identifies "disappearance of the customer during the product registration operation".

[0124] In the above embodiment, the information processing device 22 may include a recording unit that records image data acquired from each camera 21. For example, if a customer's disappearance or appearance is recognized, the image data acquired by the recording unit may be played back to verify the disappearance or appearance.

[0125] In addition, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the invention and its equivalents as described in the claims. The invention described in the original claims of this application is listed below. [Note 1] Operation recognition means for recognizing operations performed by the operator on the payment terminal, Action recognition means for recognizing the actions of the operator performing operations on the payment terminal, If, before the operation recognition means recognizes that the operation on the payment terminal is a payment completion operation, the action recognition means recognizes that the operator has left the payment terminal, the notification means provides notification to that effect. An information processing device equipped with the following. [Note 2] The notification means shall notify the operator if an appearance action is recognized in which the operator appears at the payment terminal within a predetermined time after the disappearance action is recognized by the action recognition means. The information processing device described in Appendix 1. [Note 3] The notification means shall not notify the fact that, before the operation recognition means recognizes that the operation to the payment terminal is a payment completion operation, an operation to cancel the operation to the payment terminal is recognized, and the action recognition means recognizes the disappearance action. The information processing device described in Appendix 1. [Note 4] The operation recognition means recognizes the operation performed by the operator on the payment terminal based on the images that transition in response to the operation on the payment terminal. An information processing device as described in any one of the items 1 or 2 of the appendix. [Note 5] An imaging device for capturing images of the operator performing operations on the payment terminal, An information processing device comprising: operation recognition means for recognizing operations performed by the operator on the payment terminal; action recognition means for recognizing the operator's actions based on image data of the operator captured by the imaging device; and notification means for notifying the operator if the action recognition means recognizes that the operator has left the payment terminal before the operation recognition means recognizes that the operation on the payment terminal is a payment completion operation. An information processing system equipped with the following features. [Note 6] The computer of the information processing device, Operation recognition means for recognizing operations performed by an operator on a payment terminal, Action recognition means for recognizing the actions of the operator performing operations on the payment terminal, and If, before the operation recognition means recognizes that the operation on the payment terminal is a payment completion operation, the action recognition means recognizes that the operator has left the payment terminal, a notification means provides notification to that effect. A control program to enable it to function as such. [Explanation of symbols]

[0126] 11...Self-service POS terminal, 12...POS server, 13...Display control device, 14...Attendant terminal, 15...Communication network, 21...Camera, 22...Information processing device, 40...Main unit, 41...Touch panel, 42...Reading window, 43...Card slot, 44...Receipt printing slot, 45...Coin slot, 46...Coin dispensing slot, 47...Banknote slot, 48...Banknote dispensing slot, 50...Housing, 51...Scale unit, 52...Bag holder, 53...Holding arm, 54...Weighing pan, 55...Placement surface, 60...Basket stand, 61...Communication cable, 62...Reader / Writer, 63...Stand, 64...Display Pole, 65...Light-emitting Unit, 81...Processor, 82...Main Memory, 83...Auxiliary Storage Device, 84...Clock, 85...Timer, 86...Camera Interface, 87...Communication Interface, 88...System Bus, 100...Self-POS System, 200...Information Processing System, 821...Management Table, 822...Message Table, 823...Time-Series Buffer, 221...Action Recognition Unit, 222...Operation Recognition Unit, 223...Determination Unit, 224...Notification Unit.

Claims

1. Action recognition means for recognizing the appearance action of a person appearing at a payment terminal and the disappearance action of the person leaving the payment terminal, When the behavior is recognized by the behavior recognition means, the assigning means assigns identification information to identify the person in question as the operator. Operation recognition means for recognizing operations on the payment terminal, A recording means that records the operation recognized by the operation recognition means as the operation of an operator identified by the identification information assigned by the assignment means, If the disappearance behavior is recognized by the behavior recognition means and the payment completion operation as an operation to the payment terminal is not recorded by the recording means, the notification means provides a notification to prompt the attendant to confirm that the operator of the payment terminal disappeared during the payment operation, and if the reappearance behavior is recognized within a predetermined time after the disappearance behavior is recognized, the notification means provides a notification to prompt the attendant to confirm that the operator has reappeared. An information processing device equipped with the following.

2. The notification means does not provide the notification if, even if the settlement completion operation is not recorded by the recording means, the cancellation operation to stop the operation to the settlement terminal is recorded by the recording means. The information processing apparatus according to claim 1.

3. An imaging device that captures images of the operator performing operations on the payment terminal, Action recognition means that recognizes the appearance action of a person appearing at a payment terminal and the disappearance action of the person leaving the payment terminal, based on the imaging data captured by the imaging device. When the behavior is recognized by the behavior recognition means, the assigning means assigns identification information to identify the person in question as the operator. Operation recognition means for recognizing operations on the payment terminal, A recording means for recording the operation recognized by the operation recognition means as the operation of an operator identified by the identification information assigned by the assignment means, and If the disappearance behavior is recognized by the behavior recognition means and the payment completion operation as an operation to the payment terminal is not recorded by the recording means, the notification means provides a notification to prompt the attendant to confirm that the operator of the payment terminal disappeared during the payment operation, and if the reappearance behavior is recognized within a predetermined time after the disappearance behavior is recognized, the notification means provides a notification to prompt the attendant to confirm that the operator has reappeared. An information processing device equipped with, An information processing system equipped with the following features.

4. The computer of the information processing device, Action recognition means for recognizing the appearance action of a person appearing at a payment terminal and the disappearance action of the person leaving the payment terminal. When the behavior is recognized by the behavior recognition means, the assigning means assigns identification information to identify the person in question as the operator. Operation recognition means for recognizing operations on the payment terminal, A recording means for recording the operation recognized by the operation recognition means as the operation of an operator identified by the identification information assigned by the assignment means, If the disappearance behavior is recognized by the behavior recognition means and the payment completion operation as an operation to the payment terminal is not recorded by the recording means, the notification means provides a notification to prompt the attendant to confirm that the operator of the payment terminal disappeared during the payment operation, and if the reappearance behavior is recognized within a predetermined time after the disappearance behavior is recognized, the notification means provides a notification to prompt the attendant to confirm that the operator has reappeared. A control program to enable it to function as such.

Images