Monitoring device, settlement device, and storage medium

Abstract

A monitoring device, a settlement device, and a storage medium capable of suppressing improper behavior of a customer toward a settlement device of a full self-service are disclosed. The monitoring device includes a behavior recognition section, an operation recognition section, an improper detection section, an image acquisition section, and an output section. The behavior recognition section recognizes behavior of a settler toward the settlement device. The operation recognition section recognizes operation input to the settlement device by the settler. The improper detection section detects improper behavior of the settler based on recognition results recognized by the behavior recognition section and the operation recognition section. The image acquisition section acquires an image specifying improper behavior detected by the improper detection section from a photographic image of a camera that photographs the settler operating the settlement device. The output section outputs the image specifying the improper behavior.

Description

[0001] This application claims priority from Japanese Application No. JP 2022-020710 filed February 14, 2022, and the contents of the above application are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to a monitoring device, a settlement device, and a storage medium for each. BACKGROUND

[0003] In recent years, in retail stores such as supermarkets, from the viewpoint of reduction of labor costs, countermeasures against infection of infectious diseases, and the like, a self-service POS (Point Of Sales) terminal has been attracting attention. The self-service POS terminal is a settlement device in which a customer himself / herself performs all self-service operations from registration to settlement of a purchased product. Therefore, development of a technology capable of suppressing improper behavior of a customer is sought. SUMMARY

[0004] In view of the above problem, the present application aims to provide a monitoring device, a settlement device, and a storage medium for each, which suppress improper behavior of a customer with respect to a settlement device for all self-service operations.

[0005] To solve the above problem, the monitoring device of the present application includes a behavior recognition unit, an operation recognition unit, an improper detection unit, an image acquisition unit, and an output unit. The behavior recognition unit recognizes behavior of a settlement person with respect to a settlement device. The operation recognition unit recognizes operation input performed by the settlement person to the settlement device. The improper detection unit detects improper behavior of the settlement person based on recognition results recognized by the behavior recognition unit and the operation recognition unit. The image acquisition unit acquires an image specifying the improper behavior detected by the improper detection unit from a photographic image of a camera that photographs the settlement person operating the settlement device. The output unit outputs the image specifying the improper behavior.

[0006] According to the above monitoring device, a monitoring device capable of suppressing improper behavior of a customer with respect to a settlement device for all self-service operations can be provided.

[0007] In the above monitoring device, the behavior recognition unit recognizes behavior of the settlement person based on the photographic image of the camera.

[0008] According to the above monitoring device, behavior recognition can be achieved based on recognition from a photographic image.

[0009] In the above-described monitoring device, the behavior recognition section recognizes at least a taking-out behavior in which the customer takes a product in hand and a bagging behavior in which the customer bags a product held in hand, the operation recognition section recognizes at least an operation input for product registration to the settlement device, the irregularity detection section detects the irregularity when the operation recognition section does not recognize the operation input for the product registration during the taking-out behavior and the bagging behavior recognized by the behavior recognition section, and the image acquisition section detects an image containing a product that has been bagged by the bagging behavior from a captured image of the camera used in the bagging behavior of the customer.

[0010] According to the above-described monitoring device, the irregularity can be detected from a series of behaviors recognized by the behavior recognition section.

[0011] In the above-described monitoring device, the behavior recognition section recognizes at least a registration behavior in which the customer registers a product held in hand to the settlement device, the operation recognition section recognizes at least an operation input for product registration to the settlement device, the irregularity detection section detects the irregularity when the operation recognition section cannot recognize the operation input for the product registration although the registration behavior is recognized by the behavior recognition section, and the image acquisition section detects an image containing a product aligned to the settlement device by the registration behavior from a captured image of the camera used in the registration behavior of the customer.

[0012] According to the above-described monitoring device, the product can be specified from the video regarding the irregularity.

[0013] In the above-described monitoring device, the output section displays an image specifying the irregularity on a display device of the settlement device.

[0014] According to the above-described monitoring device, the irregularity recognition image can be displayed together with the warning message.

[0015] In the above-described monitoring device, the output section further displays an image specifying the irregularity on a display device of a clerk terminal used by a clerk.

[0016] According to the above-described monitoring device, the irregularity recognition image and the warning message can be displayed on the display device of the clerk terminal.

[0017] In the above-described monitoring device, a threshold value for determining whether or not to notify the clerk terminal is set, and the output section displays an image specifying the irregularity on a display device of the clerk terminal when the number of times of the irregularity detected by the irregularity detection section reaches the threshold value.

[0018] According to the above-described monitoring device, it is possible to adjust the frequency of displaying the improper identification image and the warning message on the display device of the attendant terminal according to the setting of the threshold value.

[0019] The storage medium of another aspect of the present application stores a program for causing a computer to function as: a behavior recognition section that recognizes a behavior of a person who performs a settlement with a settlement device; an operation recognition section that recognizes an operation input performed by the person to the settlement device; an improper detection section that detects an improper behavior of the person based on recognition results recognized by the behavior recognition section and the operation recognition section; an image acquisition section that acquires an image that specifies the improper behavior detected by the improper detection section from a photographic image of a camera that photographs the person who performs the settlement with the settlement device; and an output section that outputs the image that specifies the improper behavior.

[0020] According to the above-described storage medium, it is possible to provide a computer with a function of suppressing an improper behavior of a customer with respect to a settlement device of a full self-service.

[0021] The settlement device of another aspect of the present application includes: a behavior recognition section that recognizes a behavior of a person who performs a settlement; an operation recognition section that recognizes an operation input performed by the person; an improper detection section that detects an improper behavior of the person based on recognition results recognized by the behavior recognition section and the operation recognition section; an image acquisition section that acquires an image that specifies the improper behavior detected by the improper detection section from a photographic image of a camera that photographs the person; and an output section that outputs the image that specifies the improper behavior.

[0022] According to the above-described settlement device, it is possible to provide a settlement device that suppresses an improper behavior of a customer.

[0023] The storage medium of another aspect of the present application stores a program for causing a computer to function as: a behavior recognition section that recognizes a behavior of a person who performs a settlement; an operation recognition section that recognizes an operation input performed by the person; an improper detection section that detects an improper behavior of the person based on recognition results recognized by the behavior recognition section and the operation recognition section; an image acquisition section that acquires an image that specifies the improper behavior detected by the improper detection section from a photographic image of a camera that photographs the person; and an output section that outputs the image that specifies the improper behavior.

[0024] According to the above-described storage medium, it is possible to provide a computer with a function of suppressing an improper behavior of a customer. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a schematic configuration diagram of a full self-service type checkout system;

[0026] Figure 2 It is a three-dimensional diagram showing the appearance of a self-service POS terminal;

[0027] Figure 3 It is a block diagram showing the main circuitry of the monitoring device;

[0028] Figure 4 This is a schematic diagram illustrating the data structure of an image storage device;

[0029] Figure 5 This is a schematic diagram representing the data structure of the state memory;

[0030] Figure 6 This is a schematic diagram illustrating the data structure of a list storage device;

[0031] Figure 7 This is a schematic diagram illustrating the data structure of improperly stored memory;

[0032] Figure 8 This is a flowchart illustrating the function of the operation recognition unit;

[0033] Figure 9 This is a flowchart used to explain the function of the behavior recognition unit;

[0034] Figure 10 This is a flowchart illustrating the functions of the improper detection unit, the image acquisition unit, and the output unit;

[0035] Figure 11 This is a flowchart illustrating the functions of the improper detection unit, the image acquisition unit, and the output unit;

[0036] Figure 12 This is a flowchart illustrating the functions of the improper detection unit, the image acquisition unit, and the output unit;

[0037] Figure 13 This is a flowchart illustrating the functions of the improper detection unit, the image acquisition unit, and the output unit;

[0038] Figure 14 This is a flowchart illustrating the functions of the improper detection unit, the image acquisition unit, and the output unit;

[0039] Figure 15 This is an illustration showing an example of a product registration screen; and

[0040] Figure 16 This is a schematic diagram illustrating an example of the screen displayed on a server's terminal.

[0041] Explanation of reference numerals in the attached figures

[0042] 10 Self-service POS terminals, 11 Touch panels, 14 Scanners

[0043] 20 Waiter terminals; 30 Store servers; 40 Communication networks

[0044] 50 Monitoring device 51 Processor 52 Main memory

[0045] 53 Auxiliary storage device 54 Clock 55 Camera interface

[0046] 56 Terminal interface 57 Network interface 60 Camera

[0047] 100 Checkout System 102 First Product Display Stand 103 Second Product Display Stand

[0048] 511 Operation Recognition Department; 512 Behavior Recognition Department; 513 Improper Detection Department

[0049] 514 Image acquisition unit; 515 Output unit; 521 Image memory

[0050] 522 State memory; 523 List memory; 524 Improper memory

[0051] Qa and Qb improperly specify the image display area. Detailed Implementation

[0052] The following description uses the accompanying drawings to illustrate the embodiments.

[0053] [Explanation of the Checkout System]

[0054] Figure 1 This is a schematic diagram illustrating the overall structure of a fully self-service checkout system 100. The checkout system 100 includes multiple self-service POS terminals 10, server terminals 20, and a store server 30. The checkout system 100 connects the self-service POS terminals 10, server terminals 20, and store server 30 via a communication network 40, such as a LAN (Local Area Network). The store server 30 provides various services related to this checkout system 100 to the self-service POS terminals 10 and server terminals 20 connected via the communication network 40.

[0055] Multiple self-service POS terminals 10 are pre-installed at the store's checkout area, and customers who have finished shopping are the operators. In a sales area where goods are displayed, customers who place their purchased items in shopping baskets or similar containers and move towards the checkout area operate the self-service POS terminals 10 to independently complete the transaction involving registration and payment. The self-service POS terminal 10 is an example of a settlement device. The settlement device can be referred to as a settlement terminal, checkout device, checkout terminal, merchandise registration device, or merchandise registration terminal, etc.

[0056] A server terminal 20 is installed in a monitoring area near the exit of the checkout area. The server terminal 20 displays a list of purchased items registered at each self-service POS terminal 10 on its display device. Furthermore, if an error occurs at a self-service POS terminal 10, the server terminal 20 notifies the customer of the error. Store clerks, or servers, in the monitoring area monitor the checkout area based on the information displayed on or provided to them by the server terminal 20. Additionally, although... Figure 1 The number of waiter terminals 20 is one, but multiple waiter terminals 20 can also be connected to the communication network 40.

[0057] Each self-service POS terminal 10 includes a monitoring device 50. The monitoring device 50 can be located either outside or inside the self-service POS terminal 10. The monitoring device 50 is used to monitor customers operating the self-service POS terminal 10, also known as checkout customers. Specifically, the monitoring device 50 monitors checkout customers operating the self-service POS terminal 10 and making transactions based on information such as images captured by a camera 60 located near the self-service POS terminal 10, and issues an alarm if the checkout customer engages in improper conduct. Improper conduct includes, for example, bagging purchased goods without registering them in the self-service POS terminal 10. Improper conduct also includes, for example, leaving the checkout area without paying for the purchased goods. The alarm notification is sent to either the self-service POS terminal 10 or the server terminal 20. Each monitoring device 50 is pre-connected to the server terminal 20 via a communication network 40 and can output alarm signals to the server terminal 20. Alternatively, each monitoring device 50 can be connected to the server terminal 20 without going through the communication network 40.

[0058] [Instructions for Self-Service POS Terminals]

[0059] Figure 2 This is a perspective view showing the external configuration of the self-service POS terminal 10. The self-service POS terminal 10 includes a main body 101, and a first merchandise display stand 102 and a second merchandise display stand 103 disposed on the left and right sides of the main body 101. The first merchandise display stand 102, facing the main body 101 and disposed on the right side, is a place for customers, as checkout users, to place goods purchased before registration. Goods purchased before registration are usually placed in a shopping basket and placed on the first merchandise display stand 102. Goods purchased before registration can also be placed on top of the first merchandise display stand 102 while in a shopping cart.

[0060] The second merchandise display stand 103, facing the main body 101 and positioned on the left, is where the checkout person places the registered purchased goods. The registered goods are placed in containers, such as shopping baskets, shopping bags, or reusable bags, on the second merchandise display stand 103. The registered goods can also be placed directly on the second merchandise display stand 103. Figure 2 In this design, a temporary placement table 104 is mounted on the upper part of the second merchandise display table 103 via a support column. The temporary placement table 104 is used, for example, for temporarily placing merchandise before putting it into shopping bags, etc. Two holding arms 105 are provided on the temporary placement table 104, spaced apart from each other on the left and right. By using these holding arms 105, checkout customers can place shopping bags, reusable bags, etc., in an open state on the second merchandise display table 103.

[0061] The main body 101 includes a touch panel 11, a card reader 12, a printer 13, a scanner 14, a handheld scanner 15, a cash processor 16, and a speaker 17 as input / output devices.

[0062] The touch panel 11 includes a display as a display device and a touch sensor as an input device. The display shows various screen information such as the checkout start screen, the product registration screen, the payment method selection screen, and the checkout screen. The checkout start screen is the screen where the transaction begins. A start button is configured on the checkout start screen to indicate the start of the transaction. The person registering to purchase goods touches the start button. When the touch sensor detects that the start button has been touched, the controller of the self-service POS terminal 10 switches the display screen from the checkout start screen to the product registration screen.

[0063] The product registration screen displays detailed information such as the product name, price, and total amount of the registered purchase items. A checkout button is included on the product registration screen to indicate the transition to payment for the completed purchase. The customer who has completed the purchase touches the payment button. When the touch sensor detects the checkout button being touched, the controller of the self-service POS terminal 10 switches the display from the product registration screen to the payment method selection screen.

[0064] The payment method selection screen accepts input for payment methods such as cash, credit card, and electronic money. This screen includes buttons for cash payments, credit card payments, and electronic money payments. For example, a customer wishing to pay in cash touches the cash button. When the touch sensor detects the touch, the controller of the self-service POS terminal 10 switches the display from the payment method selection screen to the checkout screen. The checkout screen displays the total transaction amount, the prepaid cash amount, and the change given.

[0065] Card reader 12 is a device for reading data stored on card media such as credit cards, electronic money cards, and loyalty cards. When the card media is a magnetic stripe card, card reader 12 is a magnetic stripe card reader. When the card media is an IC card, card reader 12 is an IC card reader. Self-service POS terminal 10 may include either a magnetic stripe card reader or an IC card reader, or both, as card reader 12. Card reader 12 may also be a card reader / writer with the function of writing data to the card.

[0066] Printer 13 is a device that prints data related to purchase receipts, credit card receipts, etc., onto receipt paper. The receipt paper printed with various data by printer 13 is cut by a cutter and dispensed from the receipt dispensing slot. Printer 13 is implemented using, for example, a thermal printer or a click-to-print printer.

[0067] Scanner 14 and handheld scanner 15 are devices for reading and identifying product information. In this embodiment, scanner 14 and handheld scanner 15 are devices for reading barcodes attached to products as information for identifying products. Scanner 14 optically reads the product barcode aligned with the reading window (glass window) in a manner opposite to the barcode attached to the product. Handheld scanner 15 is a device operated by the checkout person by holding it in their hand and optically reading the barcode attached to the product by bringing it close to the checkout. Furthermore, scanner 14 and handheld scanner 15 can be any scanner that reads and identifies product information; for example, they can also be scanners that read information other than barcodes, such as QR codes.

[0068] Cash processing machine 16 is a device for processing cash. Cash processing machine 16 has a banknote input slot 161, a banknote discharge slot 162, a coin input slot 163, and a coin discharge slot 164. Cash processing machine 16 processes banknotes inserted into banknote input slot 161. Cash processing machine 16 discharges banknotes as change from banknote discharge slot 162. Cash processing machine 16 accepts and processes coins inserted into coin input slot 163. Cash processing machine 16 discharges coins as change from coin discharge slot 164.

[0069] Speaker 17 outputs a warning tone or a voice message.

[0070] A camera 60, serving as a recording device, is mounted above the main body 101. The camera 60 captures images of the user operating the self-service POS terminal 10. Figure 2As shown, a touch panel 11 display, a card reader 12 card slot, a printer 13 ticket dispensing slot, a scanner 14 reading window, and a cash handling machine 16 banknote insertion slot 161, banknote outlet 162, coin insertion slot 163, and coin outlet 164 are arranged on one side of the main body 101 (hereinafter referred to as the front). Therefore, the clerk stands facing the main body 101 and operates the self-service POS terminal 10. A camera 60 captures a roughly full-body image of the clerk from above. Thus, the camera 60 captures the clerk's head, face, shoulders, arms, chest, abdomen, etc.

[0071] [Composition Description of the Monitoring Device]

[0072] Figure 3 This is a block diagram showing the main circuitry of the monitoring device 50. The monitoring device 50 includes a processor 51, main memory 52, auxiliary storage device 53, clock 54, camera interface 55, terminal interface 56, network interface 57, and system transmission line 58. The system transmission line 58 includes an address bus, data bus, and control signal lines. The system transmission line 58 connects the processor 51 and other components, transmitting data signals exchanged between them.

[0073] The monitoring device 50 is connected to the processor 51, main memory 52, auxiliary storage device 53, clock 54, camera interface 55, terminal interface 56 and network interface 57 through the system transmission line 58 to form a computer.

[0074] Processor 51 is equivalent to the central part of the aforementioned computer. Processor 51 controls the various parts that perform the functions of the monitoring device 50 according to the operating system or application program. Processor 51 is, for example, a CPU (Central Processing Unit).

[0075] Main memory 52 corresponds to the main storage portion of the computer described above. Main memory 52 includes non-volatile memory areas and volatile memory areas. Main memory 52 stores the operating system or application programs in the non-volatile memory areas. Main memory 52 sometimes also stores data required by the processor 51 for controlling various parts during processing in both non-volatile and volatile memory areas. Main memory 52 uses the volatile memory areas as working areas where data can be appropriately rewritten by the processor 51. Non-volatile memory areas are, for example, ROM (Read Only Memory). Volatile memory areas are, for example, RAM (Random Access Memory).

[0076] Auxiliary storage device 53 corresponds to the auxiliary storage section of the computer described above. For example, EEPROM (Electric Erasable Programmable Read-Only Memory), HDD (Hard Disc Drive), or SSD (Solid State Drive) constitutes auxiliary storage device 53. Auxiliary storage device 53 stores data used by processor 51 during various processing operations, as well as data generated by processing performed by processor 51. Auxiliary storage device 53 may also sometimes store the aforementioned application programs.

[0077] Camera interface 55 connects to camera 60 installed in self-service POS terminal 10. Camera interface 55 outputs control signals to camera 60 to control the camera's operation. Camera 60 performs photographic actions according to these control signals. Camera interface 55 acquires image data captured by camera 60.

[0078] Terminal interface 56 connects to self-service POS terminal 10. Data communication occurs between terminal interface 56 and self-service POS terminal 10.

[0079] Network interface 57 is connected to waiter terminal 20 via communication network 40. Network interface 57 performs data communication with waiter terminal 20.

[0080] The monitoring device 50 configured accordingly uses a portion of the volatile memory area of ​​the main memory 52 as an area for image memory 521, status memory 522, list memory 523, and improper memory 524.

[0081] Figure 4 This is a schematic diagram illustrating the data structure of the image memory 521. The image memory 521 is an area used to store multiple frame images captured by the camera 60 at a time associated with the moment of photography.

[0082] Figure 5 This is a schematic diagram illustrating the data structure of the state memory 522. The state memory 522 is an area used to store multiple state data associated with transition times. The state data represents various states related to the actions of the payer operating the self-service POS terminal 10 and the input operations performed by the payer to the self-service POS terminal 10. Details regarding the states will be described later. A transition time is the moment when the state changes to the state represented by the state data.

[0083] Figure 6This is a schematic diagram illustrating the data structure of the list storage 523. List storage 523 is an area used to store multiple registered product data associated with the registration time. Registered product data refers to the data related to products already registered on the self-service POS terminal 10, including product names and product images. Registered product data may further include data such as price and quantity purchased. Product images may also be omitted from the registered product data.

[0084] Figure 7 This is a schematic diagram illustrating the data structure of the illegitimate memory 524. The illegitimate memory 524 is a region used to store multiple illegitimate data items associated with the identification time. The illegitimate data includes illegitimate distinctions, frequency, and illegitimate identification images. Illegitimate distinctions are information that identifies the type of illegitimate behavior. Illegitimate types will be described later. Frequency refers to the number of times the illegitimate behavior occurred. The illegitimate identification image is an image that can specify the illegitimate behavior.

[0085] Return to Figure 3 Explanation.

[0086] The processor 51 includes at least the functions of an operation recognition unit 511, a behavior recognition unit 512, an improper behavior detection unit 513, an image acquisition unit 514, and an output unit 515. The operation recognition unit 511 is responsible for recognizing operation inputs made by the POS terminal 10 by the payer. The operation recognition unit 511 can be referred to as an operation recognition means. The behavior recognition unit 512 is responsible for recognizing the payer's behavior towards the self-service POS terminal 10. The behavior recognition unit 512 can be referred to as a behavior recognition means. The improper behavior detection unit 513 is responsible for detecting improper behavior by the payer based on the recognition results of the behavior recognition unit 512 and the operation recognition unit 511. The improper behavior detection unit 513 can be referred to as an improper behavior detection means. The image acquisition unit 514 is responsible for acquiring images of improper behavior detected by the improper behavior detection unit 513 from photographic images taken by the camera 60 of the payer operating the self-service POS terminal 10. The image acquisition unit 514 can be referred to as an image acquisition means. The output unit 515 is designed to output an image of the specified improper behavior to the self-service POS terminal 10 and / or the staff terminal 20. The output unit 515 can be referred to as an output means.

[0087] The functions of the operation recognition unit 511, behavior recognition unit 512, improper detection unit 513, image acquisition unit 514, and output unit 515 are realized by the processor 51 through information processing performed in accordance with the accountant monitoring program.

[0088] The billing monitor program is an application program stored in the main memory 52 or the secondary storage device 53. There is no particular limitation on the method of installing the billing monitor program in the main memory 52 or the secondary storage device 53. The billing monitor program can be stored in a removable storage medium, or distributed via network communication and installed in the main memory 52 or the secondary storage device 53. The storage medium is not limited in form, as long as it is capable of storing programs and can be read by the device, such as a CD-ROM or memory card.

[0089] [Functional Description of the Monitoring Device]

[0090] Next, use Figure 8 to Figure 14 Flowchart and Figure 15 , Figure 16 The screen example describes the functions of the operation recognition unit 511, behavior recognition unit 512, improper detection unit 513, image acquisition unit 514, and output unit 515 included in the processor 51 of the monitoring device 50. Furthermore, the order and content of the flowcharts described below are just one example. The order and content can be appropriately changed as long as the same effect is achieved. Similarly, the screen example is also just one example. The layout, displayed items, etc., can be appropriately changed as long as the same effect is achieved.

[0091] Figure 8 This is a flowchart illustrating the function of the operation recognition unit 511. The customer, having finished shopping and entered the checkout area, stands in front of an available self-service POS terminal 10. The customer places a shopping basket containing the purchased items onto the first merchandise display stand 102, and places empty shopping baskets, shopping bags, reusable bags, etc., onto the second merchandise display stand 103. A checkout start screen is displayed on the touch panel 11 of the self-service POS terminal 10. The customer touches the start button located on the checkout start screen.

[0092] As Figure 8 In ACT1, the processor 51 of the monitoring device 50 waits to receive operation input data for starting use. This operation input data is output from the self-service POS terminal 10 to the monitoring device 50 when the start button on the checkout start screen is touched. When operation input data for starting use is input, the processor 51 proceeds to ACT2. In ACT2, the processor 51 initializes the image memory 521, state memory 522, list memory 523, and improper memory 524. In ACT3, the processor 51 obtains the time indicated by the clock 54 as the transition time and stores state "11" in the state memory 522 in association with this transition time. That is, state "11" indicates that the operation input for starting use by the checkout user has been recognized to the self-service POS terminal 10.

[0093] As ACT4, processor 51 notifies the start of behavior recognition. This notification is made to processor 51 itself. Upon receiving the notification, processor 51 begins to function as behavior recognition unit 512. That is, processor 51 executes the function of behavior recognition unit 512 in parallel with its function as operation recognition unit 511. Here, for ease of explanation, the function of behavior recognition unit 512 will be explained first.

[0094] Figure 9 This is a flowchart illustrating the functions of the behavior recognition unit 512. When the behavior recognition unit 512 begins functioning, as ACT21, the processor 51 outputs a start signal to the camera 60. The camera 60 accepts the start signal and begins taking pictures. As ACT22, the processor 51 acquires a photographic image based on the camera 60. The photographic image can be a still image or a video. As ACT23, the processor 51 recognizes the behavior of the person facing the self-service POS terminal 10 based on the photographic image. Specifically, the processor 51 infers the skeleton of the person facing the self-service POS terminal 10 based on the photographic image. Skeletal inference is performed using AI technologies such as deep learning. The processor 51 identifies actions such as taking out, registering, bagging, or exiting based on hand movements obtained through, for example, skeletal inference. Furthermore, the function of the behavior recognition unit 512 can also be implemented based on motion detection or the detection of the operator's movements.

[0095] The act of taking something out refers to the act of the cashier removing an item from the shopping basket placed on the first merchandise display stand 102. For example, taking an item from the first merchandise display stand 102 before registration and then putting it into the customer's pocket instead of bagging it is also a type of taking out act. The act of registering refers to the act of the cashier aligning the item with the reading window of the scanner 14 and reading the barcode attached to the item through the scanner 14. The act of bagging refers to the act of the cashier placing the item into a container prepared on the second merchandise display stand 103. In addition, placing the item on the second merchandise display stand 103 or the temporary display stand 104 without placing it in a container is also a type of bagging act. The act of leaving refers to the act of the cashier leaving in front of the self-service POS terminal 10 and exiting the checkout area.

[0096] As ACT24, processor 51 checks whether a take-out action has been identified. If no take-out action has been identified, as ACT25, processor 51 checks whether a registration action has been identified. If no registration action has been identified, as ACT26, processor 51 checks whether a bagging action has been identified. If no bagging action has been identified, as ACT27, processor 51 checks whether an exit action has been identified. If no exit action has been identified, as ACT28, processor 51 checks whether action recognition has ended. Although this will be described later, as part of the function of the operation recognition unit 511, the end of action recognition is notified. If action recognition has not ended, processor 51 returns to ACT22.

[0097] Thus, during the period before being notified that the behavior recognition has ended, processor 51 sequentially acquires photographic images (videos) based on camera 60, and recognizes the checkout customer's behavior based on these photographic images. As a result, if a take-out behavior is recognized, processor 51 proceeds to ACT 29. As ACT 29, processor 51 stores multiple frames of images showing the identified take-out behavior in image memory 521 along with their photographing times. The images showing the identified take-out behavior are images depicting the hand movements of the checkout customer as they take items from the shopping basket. As ACT 30, processor 51 obtains the time marked by clock 54 as the transition time, and stores state "21" in state memory 522 in association with this transition time. That is, state "21" represents the state where the checkout customer's take-out behavior has been recognized.

[0098] When processor 51 finishes processing ACT29 and ACT30, it returns to ACT22. Processor 51 again sequentially acquires photographic images (videos) from camera 60 and identifies the checkout customer's behavior based on these images. As a result, if registration behavior is identified, processor 51 proceeds to ACT31. In ACT31, processor 51 stores multiple frames of images showing the identified registration behavior along with their capture times in image memory 521. The images showing the identified registration behavior are those depicting the checkout customer's hand movements as they align the item with the scanner 14's reading window. In ACT32, processor 51 uses the time indicated by clock 54 as the transition time and stores state "22" in state memory 522 in association with this transition time. That is, state "22" indicates that the checkout customer's registration behavior has been identified.

[0099] When processor 51 finishes processing ACT31 and ACT32, it returns to ACT22. Processor 51 again sequentially acquires photographic images (videos) based on camera 60, and identifies the checkout person's behavior based on these images. As a result, if bagging behavior is identified, processor 51 proceeds to ACT33. In ACT33, processor 51 stores multiple frames of images that identify bagging behavior along with their capture times in image memory 521. The images that identify bagging behavior are images showing the checkout person's hand movements when placing the goods into the container. Alternatively, they are images showing the checkout person's hand movements when placing the goods on the second goods placement table 103 or the temporary placement table 104. In ACT34, processor 51 obtains the time marked by clock 54 as the transition time, and stores state "23" in state memory 522 in association with this transition time. That is, state "23" indicates that the checkout person's bagging behavior has been identified.

[0100] When processing ACT33 and ACT34 is completed, processor 51 returns to ACT22. Processor 51 again sequentially acquires photographic images (videos) based on camera 60, and identifies the checkout user's behavior based on these images. As a result, if exit behavior is identified, processor 51 proceeds to ACT35. In ACT35, processor 51 stores multiple frames of images showing the exit behavior along with their capture times in image memory 521. The images showing the exit behavior are those depicting the checkout user's actions as they leave in front of the self-service POS terminal 10. In ACT36, processor 51 obtains the time marked by clock 54 as the transition time, and stores state "24" in state memory 522 in association with this transition time. That is, state "24" indicates that the checkout user's exit behavior has been identified.

[0101] When processor 51 finishes processing ACT35 and ACT36, it returns to ACT22. Processor 51 again sequentially acquires photographic images (videos) based on camera 60, and identifies the cashier's behavior based on these images. Furthermore, if the cashier's actions are identified as taking out, registering, bagging, or exiting, processor 51 executes the processing of ACT29 to ACT36 in the same manner as described above.

[0102] When the behavior recognition is notified to be complete, processor 51 proceeds to ACT37. In ACT37, processor 51 outputs a stop signal to camera 60. Camera 60 accepts the stop signal and stops recording. At this point, processor 51 ceases its function as the behavior recognition unit 512.

[0103] Return to the function description of the operation identification unit 511.

[0104] When the payer touches the start button on the checkout start screen, the screen on touch panel 11 switches from the checkout start screen to the product registration screen (see reference). Figure 15 ).

[0105] Figure 15 This is an example of a product registration screen SCa. As shown in the figure, the product registration screen SCa includes a message display area Ma, a list display area La, a total display area Aa, a product registration image display area Pa, an improperly specified image display area Qa, a checkout button Bx, a stop button By, and a staff call button Bz.

[0106] In the message display area Ma, a message indicating the operation method for registering goods is displayed, such as "Please bring the product barcode close to the reader window." In the list display area La, the product name, quantity, and price of the registered goods are displayed in a list format. In the total display area Aa, the total number of registered goods and the total amount are displayed. In the registered goods image display area Pa, an image of the goods registered as purchased goods is displayed. The improper designation image display area Qa displays an image and warning message obtained through the image acquisition unit 514 for designating improper behavior. The checkout key Bx is a soft key used by the checkout operator to indicate the start of checkout after the registration of purchased goods has ended. The abort key By is a soft key used by the checkout operator to indicate the abort of the registration of purchased goods. The cashier call key Bz is a soft key used by the checkout operator to instruct the cashier to call the cashier.

[0107] Figure 16 It is relative to the display. Figure 15 This is an example of a product registration screen SCa displayed on the display device of the self-service POS terminal 10, and a screen SCb displayed on the display device of the server terminal 20. Multiple screens SCb are displayed on the display device of the server terminal 20, each corresponding to a different product registration screen SCa from the self-service POS terminal 10.

[0108] Screen SCb includes a list display area Lb, a total display area Ab, and an improperly designated image display area Qb. The list display area Lb displays the same content as the list display area La of the product registration screen SCA. The total display area Ab displays the same content as the total display area Aa of the product registration screen SCA. The improperly designated image display area Qb displays the same content as the improperly designated image display area Qa of the product registration screen SCA. However, the warning message displayed in the improperly designated image display area Qa is intended for the checkout person, i.e., the customer, while the warning message displayed in the improperly designated image display area Qb is intended for the sales staff, i.e., the store clerk.

[0109] exist Figure 8In ACT4, the processor 51, which notifies the start of behavior recognition, waits for input data from the self-service POS terminal 10 as ACT5. This input data includes data related to product registration, input data related to registration termination instructions, and input data related to checkout start instructions. The product registration input data refers to the data of the product identified by the scanner 14 scanning the barcode after the customer aligns it with the reader window. The product data includes at least the product name. The registration termination instruction input data is the data of the self-service POS terminal 10 outputting a termination key signal after the customer touches the termination key (By) on the product registration screen (SCa). The checkout start instruction input data is the data of the self-service POS terminal 10 outputting a checkout key signal after the customer touches the checkout key (Bx) on the product registration screen (SCa).

[0110] When processor 51 inputs operation input data from self-service POS terminal 10, it performs ACT6 to confirm whether operation input data related to product registration, registration termination instruction, or checkout start instruction has been entered. If other operation input data has been entered, processor 51 returns to ACT5 to wait for the next operation input data.

[0111] While waiting to receive operation input data from ACT5 to ACT8, if operation input data related to product registration has already been entered, processor 51 proceeds to ACT9. In ACT9, processor 51 obtains the time counted by clock 54 as the registration time, and stores state "12" in the state memory 522 in association with this registration time. That is, state "12" indicates that the operation input data related to product registration has been identified.

[0112] As ACT10, processor 51 acquires the image of the registered product. For example, before the registration time, processor 51 cuts out the image of the product from multiple frame images of recognized registration actions stored in image memory 521, and acquires it as the image of the registered product. The image of the product is an image of the object that the checkout is facing the reading window. The image of the product may also include part of the checkout's hand. If the operation input data involved in product registration includes data of a pre-set product image, the registered product image may also be the product image included in the operation input data. As ACT11, processor 51 stores the product name and the image of the registered product included in the operation input data involved in product registration in list memory 523 in association with the registration time. Afterwards, processor 51 returns to ACT5, waiting to receive the next operation input data.

[0113] In the waiting states of ACT5 to ACT8 for receiving operation input data, in ACT7, if the operation input data related to the registration abort has been entered, processor 51 proceeds to ACT12. In ACT12, processor 51 obtains the time counted by clock 54 as the registration abort time, and stores state "15" in the state memory 522 in association with this registration abort time. That is, state "15" indicates that the operation input data related to the registration abort has been identified.

[0114] As ACT13, processor 51 notifies the processor 51 that behavior recognition has ended. This notification is sent to processor 51 itself. Upon receiving this notification, processor 51 terminates its function as behavior recognition unit 512. Furthermore, as ACT14, processor 51 saves the data from state memory 522 to auxiliary storage device 53. At this point, processor 51 terminates its function as operation recognition unit 511.

[0115] In the states of waiting for operation input data in ACT5 to ACT8, in ACT8, if the operation input data related to starting the checkout has been entered, processor 51 proceeds to ACT15. In ACT15, processor 51 obtains the time counted by clock 54 as the checkout start time, and stores state "13" in the state memory 522 in association with this checkout start time. That is, state "13" indicates that the operation input data related to starting the checkout has been identified.

[0116] As ACT16, processor 51 waits to receive checkout completion data. Checkout completion data is the data output from self-service POS terminal 10 when the payment for goods based on the selected payment method (cash payment, credit card payment, electronic money payment, etc.) is completed. When checkout completion data is input, processor 51 proceeds to ACT17. As ACT17, processor 51 obtains the time counted by clock 54 as the checkout completion time and stores state "14" in state memory 522 in association with this checkout completion time. That is, state "14" indicates that the operation input data involved in checkout completion has been identified.

[0117] After this, processor 51 proceeds to ACT13 as described above. That is, processor 51 notifies that the action recognition has ended. In addition, processor 51 saves the data in state memory 522 to auxiliary storage device 53. At this point, processor 51 ends its function as operation recognition unit 511.

[0118] Processor 51 executes the following whenever input data for the initial operation is entered from self-service POS terminal 10: Figure 8The processing shown in ACT1 to ACT17 is the operation recognition unit 511. Additionally, the processor 51 executes... Figure 9 The processing described in ACT21 to ACT37 is the behavior recognition unit 512. Thus, in the auxiliary storage device 53, data from a state memory 522, which sequentially stores various states involved by each payer using the self-service POS terminal 10 for self-service checkout, is stored. Therefore, by parsing the data in the state memory 522, it is possible to verify the payer's operation history and behavior history.

[0119] On the other hand, the monitoring device 50 detects improper behavior by the cashier in real time, issues a warning to the cashier, or notifies the waiter to prevent improper behavior from happening. This improper behavior prevention function is implemented by the improper behavior detection unit 513, the image acquisition unit 514, and the output unit 515.

[0120] Figure 10 to Figure 14 This is a flowchart illustrating the functions of the improper detection unit 513, the image acquisition unit 514, and the output unit 515. Figure 10 In this context, as ACT41, processor 51 waits to receive the state "11" stored in state memory 522.

[0121] When state "11" is stored in state memory 522, processor 51 proceeds to ACT42. In ACT42, processor 51 retrieves the next state data from state memory 522. Then, processor 51 waits to receive state "21" as ACT43 or state "24" as ACT44. That is, when processor 51 recognizes an operation input made by the payer to POS terminal 10 for initiating utilization, it waits to receive either the payer's withdrawal action or exit action.

[0122] In the waiting-to-accept state of ACT43 and ACT44, when state "21" is obtained, processor 51 proceeds to ACT45. In ACT45, processor 51 retrieves the next state data from state memory 522. Thereafter, processor 51 waits to accept state "22" as ACT46, state "23" as ACT47, or state "24" as ACT48. That is, when processor 51 identifies a customer's withdrawal action, it waits to accept identification of the customer's registration action, bagging action, or exit action.

[0123] In the waiting accept state from ACT46 to ACT48, when state "22" is acquired, processor 51 proceeds to... Figure 11ACT51. As ACT51, processor 51 retrieves the next state data from state memory 522. Afterward, processor 51 waits to receive state "23" as ACT52, state "24" as ACT53, or state "21" as ACT54. That is, when processor 51 identifies a settler's registration action, it waits to receive information identifying the settler's bagging action, exit action, or retrieval action.

[0124] During the waiting acceptance states ACT52 to ACT54, when state "23" is obtained, processor 51 proceeds to ACT55. In ACT55, processor 51 retrieves a stored state one position earlier than state "23" from state memory 522. In ACT56, processor 51 checks whether the stored state one position earlier than state "23" is "12". That is, processor 51 checks whether the operation input data involved in product registration has been identified before the bagging process. If state "12" is stored, processor 51 proceeds to ACT57. In ACT57, processor 51 outputs the latest product registration image stored in list memory 523 to self-service POS terminal 10. Thus, in self-service POS terminal 10, the product registration image is displayed in the product registration image display area Pa of the product registration screen SCa. This image is an image of the purchased product registered by the checkout customer through barcode scanning by scanner 14 of self-service POS terminal 10.

[0125] Processor 51, having finished processing ACT57, returned to... Figure 10 ACT42. Afterwards, processor 51 executes the processing following ACT42 in the same manner as described above.

[0126] Thus, after recognizing the cashier's act of taking the item, followed by the registration act, identifying the operational input data involved in the item registration, and recognizing the bagging act, the processor 51 confirms that there is no improper behavior. The processor 51 then outputs an image of the registered and processed purchased item to the POS terminal 10.

[0127] Next, an explanation will be given regarding the identification of improper conduct by the settler.

[0128] First, we will explain the improper conduct of a cashier who registers the goods but proceeds with bagging them without actually registering them. This improper conduct occurs, for example, when the cashier covers the barcode with their hand and aligns it with the reader window of scanner 14. In this case, the state memory 522 stores state "22" indicating the registration action after state "21" indicating the retrieval action, followed by state "23" indicating the bagging action. However, state "12" indicating the input data for the operation related to goods registration is not stored between state "22" and state "23". That is, equivalent to... Figure 11 In ACT56, there is a case where the stored state is not "12" one step earlier than state "23" (ACT56 NO).

[0129] Processor 51 advances to Figure 12 ACT61. As ACT61, processor 51 acquires images of the bagging behavior. Specifically, before and after the transition to state "23", processor 51 acquires multiple frame images stored in image memory 521 that identify the bagging behavior. As ACT62, processor 51 cuts out images from the acquired frame images that can identify the bagged goods as the image indicating improper behavior. For example, processor 51 cuts out an image of the cashier holding the goods. For example, processor 51 cuts out an image of the cashier placing the goods on the second goods placement table 103 or the temporary placement table 104.

[0130] Here, processor 51 according to Figure 11 The processor 51 functions as an improper detection unit 513, which processes the ACT56. Additionally, the processor 51... Figure 12 The image acquisition unit 514 functions as an image acquisition unit, which processes ACT61 and ACT62.

[0131] Processor 51, having completed ACT62, proceeds to ACT63, setting the improper classification to "91". Improper classification "91" indicates information about improper conduct related to unregistered goods. As part of ACT64, processor 51 increments the count counter by 1. Figure 8 In ACT2, the count counter is reset to "0" when each memory is initialized.

[0132] As ACT65, processor 51 stores improper data in improper memory 524. That is, processor 51 uses the current moment, which is timed by clock 54, as the improper identification moment. Thereafter, processor 51 stores in improper memory 524, in association with this identification moment, the improper distinction determined in ACT63, the value of the count counter updated in ACT64, and the image of the specified improper behavior obtained in ACT62, that is, the improper identification image.

[0133] As ACT66, processor 51 outputs an improper identification image and a warning message to the payer corresponding to the improper identification "91" to self-service POS terminal 10. Thus, in self-service POS terminal 10, the improper identification image and the warning message to the payer are displayed in the improper identification image display area Qa of the product registration screen SCa. The warning message contains information such as "Product not registered. Please re-register," indicating that the product is not registered and needs to be re-registered. Therefore, for payers who have registered the product but have proceeded with packaging without registration, the improper identification image and warning message are displayed in the improper identification image display area Qa.

[0134] As ACT67, processor 51 checks whether the number of instances is greater than or equal to a threshold N. The threshold is a value used to determine whether to notify the server terminal. If the number of instances is greater than or equal to the threshold N, processor 51 proceeds to ACT68. As ACT68, processor 51 outputs an improper identification image and a warning message to the server corresponding to the improper identification "91" to the server terminal 20. Thus, in the server terminal 20, the improper identification image and the warning message to the server are displayed in the improper identification image display area Qb of the screen SCb corresponding to the self-service POS terminal 10. The warning message notifies the server of an improper act of unregistered goods, such as "an unregistered item has occurred." Therefore, if the cashier commits more than or equal to N improper acts, the server is notified via the server terminal 20.

[0135] The threshold N can be any value greater than or equal to "1". For example, if the server terminal 20 is notified of each instance of unethical behavior, the threshold N is set to "1". Alternatively, if three or more instances of unethical behavior are detected in a single transaction, the threshold N is set to "3" to notify the server terminal 20. By setting the threshold N to greater than or equal to "2", the number of times unethical behavior is notified to the server terminal 20 is reduced, thus alleviating the server's workload. However, there is a possibility that the server might miss an unethical transaction. If the threshold N is set to "1", and the server terminal 20 is notified each time unethical behavior is detected, the likelihood of the server missing unethical behavior can be reduced.

[0136] Here, the processor 51 functions as the output unit 515 through the processing of ACT66 and ACT68.

[0137] If the number of occurrences is less than the threshold N, or if the number of occurrences is greater than or equal to the threshold N, and the processor 51 outputs an improperly identified image and a warning message to the server terminal 20, then proceeds to... Figure 13ACT71. As ACT71, processor 51 retrieves the next state data from state memory 522. Afterward, processor 51 waits to receive state "22" as ACT72, state "24" as ACT73, or state "21" as ACT74. That is, when processor 51 detects improper behavior such as unregistered goods, it waits to receive registration behavior, exit behavior, or withdrawal behavior of the checkout person.

[0138] Upon confirming the improper identification image and warning message displayed in the improper designation image display area Qa of the product registration screen SCa, the checkout operator, after canceling the improper act of unregistered goods, will remove the goods designated by the improper identification image from the second product display stand 103 and perform the registration act. That is, equivalent to... Figure 13 During the waiting acceptance states of ACT72 to ACT74, state "22" was achieved (YES for ACT72).

[0139] Processor 51 proceeds to ACT75. ACT75, processor 51 removes the output of the improperly identified image and warning message. Thus, in self-service POS terminal 10, the display of the improperly designated image display area Qa is removed. In server terminal 20, the display of the improperly designated image display area Qb is removed.

[0140] When processing of ACT75 ends, processor 51 proceeds to... Figure 11 ACT51. Processor 51 performs the processing following ACT51 in the same manner as described above.

[0141] Therefore, when the barcode of the goods is read by scanner 14 through the checkout's registration action, state "12" is stored in state memory 522 after state "22". Furthermore, through the subsequent bagging action of the checkout, state "23" is stored in state memory 522. As a result, processor 51 sequentially executes processes ACT51, ACT55, and ACT57, and returns to... Figure 10 ACT42. That is, the improper conduct of not registering goods is cancelled.

[0142] On the other hand, there is a possibility that the settler might fail to cancel the improper act of not registering the goods and instead repeatedly take the next item from the first goods display stand 102. This is equivalent to... Figure 13 During the waiting acceptance state of ACT72 to ACT74, state "21" was obtained (YES for ACT74).

[0143] Processor 51 proceeds to ACT76. As ACT76, processor 51 outputs an error command to self-service POS terminal 10. The self-service POS terminal 10, having received the error command, enters an error state and becomes inoperable. The error state is not cleared unless a server-based error clearing operation is performed.

[0144] As ACT77, processor 51 outputs an error signal to server terminal 20. The error signal includes the improper identification image obtained in ACT62 and a warning message for the server corresponding to the improper distinction "91". Server terminal 20, having received the error signal, displays the improper identification image and the warning message for the server in the improper designation image display area Qb of the screen SCb corresponding to self-service POS terminal 10. Additionally, server terminal 20 informs the server that an error has occurred at self-service POS terminal 10. The method of notification is not limited. For example, it can be done by lighting a light or by sounding an error alarm. The server, having confirmed the improper content and error, alerts the cashier to go to the location of self-service POS terminal 10 and resolve the improper behavior. Furthermore, the server performs an operation to resolve the error. When the error is resolved, an error resolution signal is output from self-service POS terminal 10 to monitoring device 50.

[0145] Processor 51, which outputs erroneous commands and signals, acts as ACT78 and waits to receive an error clearance signal. When an error clearance signal is received via terminal interface 56, processor 51 returns to ACT71. Therefore, the settler cannot proceed to the next item registration unless it performs the registration of unregistered goods, that is, unless it performs an operation to clear the unregistered goods from improper registration.

[0146] Next, the explanation addresses the improper conduct of a cashier who, while registering a product, fails to bag it without registering it and instead removes the next product. Such improper conduct occurs, for example, when the cashier covers the barcode with their hand while aligning it with the scanner 14's reading window, and places the product on the ground instead of bagging it. In this case, the state memory 522 stores the state "22" representing the registration action after the state "21" representing the removal action, but does not store the state "23" representing the bagging action after that, while the state "21" representing the removal action is stored again. That is, equivalent to... Figure 11 In the waiting-to-accept states of ACT52 to ACT54, state "21" is stored (YES in ACT54).

[0147] Processor 51 advances to Figure 14ACT81. As ACT81, processor 51 retrieves a stored state one position earlier than state "21" from state memory 522. As ACT82, processor 51 confirms whether the stored state one position earlier than state "21" is "12". That is, processor 51 confirms whether the operation input data related to product registration was identified before the checkout user's retrieval action. If state "12" is stored, the unpackaged product has been correctly registered, and it is not an improper act. In this case, processor 51 proceeds to ACT83. As ACT83, processor 51 outputs the latest product registration image stored in list memory 523 to self-service POS terminal 10. Thus, in self-service POS terminal 10, the product registration image is displayed in the product registration image display area Pa of product registration screen SCa.

[0148] Processor 51, having finished processing ACT83, returned to... Figure 10 ACT45. That is, proceed to the same process as in ACT43 where state "21" was achieved. Thereafter, processor 51 executes the process following ACT45 in the same manner as described above.

[0149] On the other hand, in ACT82, if the stored state is not "12" one step earlier than state "21", the unpackaged goods are unregistered goods. In this case, processor 51 proceeds to ACT84. As ACT84, processor 51 acquires the registration behavior image. Specifically, around the time of the transition to state "22" one step earlier than state "21", processor 51 acquires multiple frame images stored in image memory 521 that identify the registration behavior. As ACT85, processor 51 cuts out an image from the acquired frame images that can identify the goods aligned with the reading window through the registration behavior as the image of the designated improper behavior. For example, processor 51 cuts out an image of the goods held by the checkout person aligned with the reading window.

[0150] Here, processor 51 according to Figure 14 The processor 512 processes the data and functions as an improper detection unit 513. Additionally, the processor 51... Figure 14 The processing of ACT84 and ACT85 functions as the image acquisition unit 514.

[0151] Processor 51, having completed ACT85, proceeds to ACT86, setting the improper classification to "92". Improper classification "92" indicates improper conduct, specifically unregistered and unpackaged goods. As ACT87, processor 51 increments the count counter by 1.

[0152] As ACT88, processor 51 stores improper data in improper memory 524. That is, processor 51 uses the current time, which is timed by clock 54, as the identification time of improper behavior. Thereafter, processor 51 stores in improper memory 524, in association with this identification time, the improper distinction determined in ACT86, the value of the count counter updated in ACT87, and the image of the specified improper behavior obtained in ACT85, i.e., the improper behavior identification image.

[0153] As ACT89, processor 51 outputs an improper identification image and a warning message to the payer corresponding to the improper identification "92" to self-service POS terminal 10. Consequently, in self-service POS terminal 10, the improper identification image and the warning message to the payer are displayed in the improper identification image display area Qa of the goods registration screen SCa. The warning message contains information such as "Goods not registered. Please re-register unpackaged goods," indicating that the goods in the image are not registered and that re-registration of unpackaged goods is required.

[0154] As ACT90, processor 51 outputs an error command to self-service POS terminal 10. Self-service POS terminal 10, having received an error command, enters an error state and becomes inoperable. The error state cannot be cleared without intervention from a service staff member.

[0155] As ACT91, processor 51 outputs an error signal to server terminal 20. The error signal includes the improper identification image obtained in ACT85 and a warning message for the server corresponding to the improper distinction "92". Server terminal 20, which receives the error signal, displays the improper identification image and the warning message for the server in the improper designation image display area Qb of the screen SCb corresponding to self-service POS terminal 10. Additionally, server terminal 20 informs the server that an error has occurred at self-service POS terminal 10. The method of notification is not limited. For example, it can be done by lighting a light or by sounding an error alarm. The server, having confirmed the improper content and error, alerts the cashier to go to the location of self-service POS terminal 10 and resolve the improper behavior. Furthermore, the server performs an operation to resolve the error. When the error is resolved, an error resolution signal is output from self-service POS terminal 10 to monitoring device 50.

[0156] Here, the processor 51 functions as the output unit 515 through the processing of ACT89 and ACT91.

[0157] As ACT92, processor 51 waits to receive an error clearing signal. Upon receiving the error clearing signal via terminal interface 56, processor 51 returns to... Figure 13ACT71. Therefore, a clearinghouse cannot register the next item unless it takes action to cancel an unpackaged, unregistered item that is not properly packaged.

[0158] Next, the improper conduct of the checkout agent in bagging items taken from the shopping basket without registering them is explained. In this case, the state memory 522 stores the state "23" representing the bagging action after the state "21" representing the taking action. That is, equivalent to... Figure 10 In the waiting acceptance states of ACT46 to ACT48, the state "23" is obtained (YES in ACT47).

[0159] Processor 51 advances to Figure 12 ACT61. Thereafter, processor 51 executes the processes ACT61 to ACT68 in the same manner as described above, and... Figure 13 The processing after ACT71 is as follows: Processor 51 acquires an image of the bagging behavior and cuts out an image specifying the improper behavior from that image. Additionally, processor 51 stores improper data in improper memory 524. Processor 51 outputs an improper identification image and a warning message to the cashier corresponding to the improper distinction "91" to the self-service POS terminal 10. Furthermore, if the number of occurrences is greater than or equal to a threshold N, processor 51 outputs an improper identification image and a warning message to the server corresponding to the improper distinction "91" to the server terminal 20.

[0160] Therefore, in the self-service POS terminal 10, in the improper designation image display area Qa of the product registration screen Sca, an image of a product that was bagged without registration and a warning message to the checkout customer are displayed as improper identification images. Thus, for checkout customers who bag products taken from the shopping basket without registration, an improper identification image and a warning message are displayed in the improper designation image display area Qa of the product registration screen Sca. Furthermore, if the number of improper acts is greater than or equal to a threshold N, the same improper identification image and a warning message to the waiter are displayed in the improper designation image display area Qb of the screen SCb corresponding to that of the self-service POS terminal 10 in the waiter terminal 20. Thus, if a checkout customer commits improper acts more than or equal to N times, the waiter is notified via the waiter terminal 20.

[0161] Subsequently, processor 51 waits to receive the next state data, obtaining state "22", state "24", or state "21". Here, if state "22" has been obtained, processor 51 deactivates the output of the improper identification image and warning message. That is, if the checkout operator corrects the improper act of not registering the item, subsequent item registration can proceed. However, if the operator takes the next item from the shopping basket without correcting the improper act, the self-service POS terminal 10 enters an error state.

[0162] Finally, we will address the improper practice of leaving the checkout without paying for the purchased goods. Such improper practice... Figure 10 The pending acceptance status of ACT43 or ACT44, or the Figure 10 The waiting status for ACT46 to ACT48, or Figure 11 The waiting acceptance status of ACT52 to ACT54 Figure 13 During the waiting acceptance states of ACT72 to ACT74, it is discovered when state "24" is obtained as the next state (YES of ACT44, ACT48, ACT54 or ACT74).

[0163] Processor 51 advances to Figure 10 ACT101. Processor 51 retrieves a stored state one position earlier than state "24" from state memory 522. As ACT102, processor 51 confirms whether the stored state one position earlier than state "24" is "14" or "15". That is, processor 51 confirms whether the operation input data involved in the checkout completion or the operation input data involved in the registration suspension was identified before the exit action. If the operation input data involved in the checkout completion or the operation input data involved in the registration suspension were identified before the exit action, the exit action is a legitimate action. Processor 51 terminates its function as the improper detection unit 513.

[0164] Correspondingly, if the stored state is not "14" or "15" one step earlier than state "24," that is, if no operation input data related to the completion of checkout or the operation input data related to the suspension of registration is identified, the exit behavior is an improper behavior. As ACT103, the processor 51 acquires the exit behavior image. Specifically, the processor 51 acquires multiple frame images stored in the image memory 521 that identify the exit behavior before and after the transition of state "24." As ACT104, the processor 51 cuts out an image that can identify the person as the checkout person from the acquired frame images as the image that specifies the improper behavior; for example, the processor 51 cuts out an image that shows the face of the checkout person.

[0165] As in ACT105, processor 51 classifies the improper behavior as "99". Improper behavior "99" indicates an improper act of exiting without checking out. As in ACT106, processor 51 increments the count counter by 1. Subsequently, as in ACT107, processor 51 stores improper data in improper memory 524. That is, processor 51 uses the current time, as measured by clock 54, as the improper identification time. Subsequently, processor 51 stores in improper memory 524 the improper behavior classification determined in ACT105, the value of the count counter updated in ACT106, and the image of the specified improper act obtained in ACT104, i.e., the improper identification image, associated with this identification time.

[0166] As ACT108, processor 51 outputs an error command to self-service POS terminal 10. Additionally, as ACT109, processor 51 outputs an error signal to server terminal 20. The error signal includes the improper identification image obtained in ACT104 and a warning message for the server corresponding to the improper distinction "99". The warning message is something that draws the server's attention, such as "There is a customer who wants to leave without paying."

[0167] The processor 51, which outputs an error command and error signal, terminates its function as the improper detection unit 513.

[0168] [Explanation of the monitoring device's effects]

[0169] As detailed above, the monitoring device 50 can effectively suppress improper behavior by customers that may occur at the self-service POS terminal 10.

[0170] For example, if a customer engages in the unauthorized purchase and packaging of goods, an image identifying the purchase and a warning message are displayed on the touch panel 11 of the self-service POS terminal 10. The unauthorized identification image is an image that identifies the unauthorized purchased goods. The warning message instructs the customer to re-register the unauthorized goods. By displaying the unauthorized identification image and the warning message together, the unauthorized behavior can be suppressed. That is, by displaying the warning message and the unauthorized identification image together on the touch panel 11, other customers and staff can be informed that unauthorized behavior has occurred. Furthermore, since the unauthorized identification image identifies the unauthorized goods, it is easy to pinpoint which item was involved in the unauthorized activity. Therefore, it is difficult for customers to engage in unauthorized behavior.

[0171] Based on this, if the checkout customer ignores the warning message and proceeds to register the next item, the self-service POS terminal 10 becomes inoperable and sends a warning to the server terminal 20. Therefore, there will be no missed instances of the aforementioned improper conduct.

[0172] For example, there may be instances where a customer makes the act of registering a purchase but does not actually bag the item, constituting an improper act. In such cases, an improper identification image and a warning message are displayed on the touch panel 11 of the self-service POS terminal 10. The improper identification image is an image that identifies the purchased item for which the customer makes the registration action. Therefore, as described above, the effect of suppressing improper behavior can be achieved.

[0173] Furthermore, if a customer engages in inappropriate behavior N times or more than the threshold, an inappropriate behavior identification image and a warning message will also be displayed on the display device of the waiter terminal 20. Therefore, the waiter can notice the inappropriate behavior and draw the customer's attention, resulting in a significant effect in suppressing inappropriate behavior.

[0174] Furthermore, if a customer leaves the checkout area without paying, an unauthorized image, such as a reflection of the customer's face, is displayed on the server terminal 20's display device. Therefore, even in a crowded checkout area with many customers, the server can easily identify customers who have improperly left.

[0175] [Variation Example]

[0176] In the above embodiments, a case is illustrated where monitoring devices 50 are provided for each of the self-service POS terminals 10. Alternatively, a monitoring device 50 may be connected to two or more self-service POS terminals 10. The processor 51 of the monitoring device 50 corresponds to each of the two or more self-service POS terminals 10 and has the functions of an operation recognition unit 511, an behavior recognition unit 512, an improper detection unit 513, an image acquisition unit 514, and an output unit 515.

[0177] Camera 60 may not be installed at the self-service POS terminal 10. For example, it may be installed on the ceiling of the checkout area. Alternatively, the behavior of the checkout person can be identified from images captured by two or more cameras.

[0178] The behavior recognition unit 512 recognizes behaviors not limited to taking out items, registering items, bagging items, and exiting the store. It can also recognize behaviors such as bagging items taken from a shopping basket without registering them, or handing items to a companion, as improper behavior. In this case, an image of the designated item can be cropped from the image of the improper behavior and output to the self-service POS terminal 10 or the server terminal 20. Alternatively, the image can be output to other terminals, such as an information communication terminal carried by the staff. Furthermore, it can be configured to recognize behaviors such as the cashier or someone other than the cashier obstructing the camera 60 with their hand, arm, etc.

[0179] In the above embodiment, the output unit 515 functions to output an improperly identified image along with a warning message, but its function is not limited to this. For example, even if the output unit 515 outputs only the improperly identified image on the self-service POS terminal 1 or the like, it can still suppress improper behavior.

[0180] In the above embodiments, an improper behavior identification image is output whenever improper behavior is detected, but the timing of image output is not limited to this. For example, the improper behavior identification image can also be output when the checkout user touches the checkout button Bx, that is, when the checkout process begins. In this case, Figure 13 If the condition in ACT74 is YES, then simply omit the processing of ACT76 to ACT78 and proceed to... Figure 10 The ACT45 processing can be used.

[0181] In the above embodiment, the processor 51 recognizes the start of use based on the touch of the start button on the checkout start screen. For example, if a human perception sensor capable of detecting a person standing in front of the self-service POS terminal 10 is provided, the processor 51 can also recognize the start of use based on the detection of a person by the human perception sensor. In this case, the processor 51 can also recognize that an exit action has been performed if the human perception sensor does not detect a person.

[0182] In the above embodiments, it was described that the monitoring device 50 can be located either outside or inside the self-service POS terminal 10. As another embodiment, the self-service POS terminal 10 can also incorporate the function of the monitoring device 50 to achieve this function. That is, it can also be a settlement device (self-service POS terminal 10) that includes a behavior recognition means (behavior recognition unit 512) for recognizing the behavior of the settler, an operation recognition means (operation recognition unit 511) for recognizing operation input based on the settler's operation, an improper detection means (improper detection unit 513) for detecting improper behavior of the settler based on the behavior recognition means and the recognition result of the operation recognition means, an image acquisition means (image acquisition unit 514) for acquiring an image of the specified improper behavior detected by the improper detection means from a photographic image taken by a camera of the settler operating the settlement device, and an output means (output unit 515) for outputting the image specifying the improper behavior.

[0183] Furthermore, although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. In addition, these embodiments and their variations are included in the scope and spirit of the invention, and are included within the scope of the invention as described in the claims and their equivalents.

Claims

1. A monitoring device, characterized in that, include: The behavior recognition unit identifies the user's behavior towards the payment device; An operation recognition unit identifies operation inputs made to the settlement device by the settler. The storage unit stores first state data and second state data in the order of the time of transition to the state represented by the first state data or the second state data. The first state data represents various states related to the behavior of the checkout user to the checkout device as identified by the behavior recognition unit, and the second state data represents various states related to the operation input made by the checkout user to the checkout device as identified by the operation recognition unit. The improper behavior detection unit detects improper behavior of the checkout operator according to the storage order of the first state data and the second state data stored by the storage unit in the order of the state transition time; the image acquisition unit acquires an image specifying the improper behavior detected by the improper behavior detection unit from the photographic image taken by the camera of the checkout operator operating the checkout device. as well as The output unit outputs an image specifying the alleged misconduct.

2. The monitoring device according to claim 1, wherein, The behavior recognition unit identifies the cashier's behavior based on the images captured by the camera.

3. The monitoring device according to claim 2, wherein, The behavior recognition unit at least recognizes the actions of the cashier taking the goods out of their hands and the actions of the cashier putting the goods into a bag. The operation recognition unit at least recognizes operation inputs used for registering goods with the checkout device. The improper detection unit detects improper behavior while storing first state data representing the state related to the taking-out behavior and first state data representing the state related to the bagging behavior, without storing second state data representing the state related to the operation input for product registration. The image acquisition unit detects images containing goods that have been bagged through the bagging process from photographs taken by the camera used to identify the bagging behavior of the checkout person.

4. The monitoring device according to claim 2, wherein, The behavior recognition unit at least recognizes the act of the payer registering the goods in their hand at the checkout device. The operation recognition unit at least recognizes operation inputs used for registering goods with the checkout device. The improper behavior detection unit detects improper behavior even when it stores first state data indicating a state related to the registration behavior but does not store second state data indicating a state related to the operation input used for product registration. The image acquisition unit detects an image containing the goods that are aligned with the checkout device during the registration process of the checkout user from the photographic image taken by the camera used to identify the checkout user's registration behavior.

5. The monitoring device according to any one of claims 1 to 4, wherein, The output unit displays an image specifying the improper conduct to the display device of the settlement device.

6. The monitoring device according to claim 5, wherein, The output unit further outputs an image display specifying the improper behavior to the display device of the waiter terminal used by the store clerk.

7. The monitoring device according to claim 6, wherein, A threshold is set to be stored for determining whether to notify the waiter terminal. When the number of times the improper behavior detected by the improper behavior detection unit reaches the threshold, the output unit displays an image specifying the improper behavior on the display device of the waiter terminal.

8. A storage medium storing programs for enabling a computer to function as: The behavior recognition unit identifies the user's behavior towards the payment device; An operation recognition unit identifies operation inputs made to the settlement device by the settler. The storage unit stores first state data and second state data in the order of the time of transition to the state represented by the first state data or the second state data. The first state data represents various states related to the behavior of the checkout user to the checkout device as identified by the behavior recognition unit, and the second state data represents various states related to the operation input made by the checkout user to the checkout device as identified by the operation recognition unit. The improper behavior detection unit detects improper behavior of the biller according to the storage order of the first state data and the second state data stored by the storage unit in the order of the state transition time; The image acquisition unit acquires an image of the improper behavior detected by the improper behavior detection unit from a photographic image taken by a camera that captures the image of the person operating the settlement device; as well as The output unit outputs an image specifying the alleged misconduct.

9. A settlement device, characterized in that, include: The behavior recognition department identifies the behavior of the payer; The operation recognition unit identifies the operation input made by the payer. The storage unit stores first state data and second state data in the order of the time of transition to the state represented by the first state data or the second state data. The first state data represents various states related to the behavior of the biller identified by the behavior recognition unit, and the second state data represents various states related to the operation inputs performed by the biller identified by the operation recognition unit. The improper behavior detection unit detects improper behavior of the biller according to the storage order of the first state data and the second state data stored by the storage unit in the order of the state transition time; The image acquisition unit acquires an image specifying the improper behavior detected by the improper behavior detection unit from the photographic image taken by the camera that captured the cashier; as well as The output unit outputs an image specifying the alleged misconduct.

10. A storage medium for storing programs that enable a computer to function as: The behavior recognition department identifies the behavior of the payer; The operation recognition unit identifies the operation input made by the payer. The storage unit stores first state data and second state data in the order of the time of transition to the state represented by the first state data or the second state data. The first state data represents various states related to the behavior of the biller identified by the behavior recognition unit, and the second state data represents various states related to the operation inputs performed by the biller identified by the operation recognition unit. The improper behavior detection unit detects improper behavior of the biller according to the storage order of the first state data and the second state data stored by the storage unit in the order of the state transition time; The image acquisition unit acquires an image specifying the improper behavior detected by the improper behavior detection unit from the photographic image taken by the camera that captured the cashier; as well as The output unit outputs an image specifying the alleged misconduct.

Images