Improper behavior identification device, storage medium, and improper behavior identification method

Abstract

The present invention provides an unfair behavior recognition device, a storage medium, and an unfair behavior recognition method, which can suppress unfair behavior of customers with respect to a settlement terminal corresponding to full self-service. The unfair behavior recognition device includes an action recognition unit, an operation recognition unit, an unfair detection unit, and a notification unit. The action recognition unit recognizes the actions of a shopper with respect to the settlement terminal. The operation recognition unit recognizes the operation performed by the shopper on the settlement terminal. The unfair detection unit detects the unfair behavior of the shopper based on the correspondence between the shopper's actions and the operation performed by the shopper on the settlement terminal. The notification unit notifies the shopper of the unfair behavior.

Description

[0001] This application claims priority from Japanese patent application filed on January 14, 2021, application number JP2021-004259, and cites the contents of the above application, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0002] The embodiments of the present invention relate to an improper behavior recognition device and storage medium, and an improper behavior recognition method. Background Art

[0003] In recent years, in retail stores such as supermarkets, self-service POS (Point Of Sales) terminals have attracted attention from the perspectives of reducing labor costs and preventing the spread of infectious diseases. Self-service POS terminals are fully self-service settlement terminals that allow customers to perform operations from registration to settlement of purchased goods by themselves. Therefore, the development of technologies that can suppress improper behavior of customers is sought. Summary of the invention

[0004] In view of the above problems, the problem to be solved by the embodiments of the present invention is to provide an unfair behavior recognition device and storage medium, as well as an unfair behavior recognition method, which can suppress customers' unfair behavior towards a fully self-service settlement terminal.

[0005] To solve the above problems, an embodiment of the present invention provides an unfair behavior recognition device including an action recognition unit, an operation recognition unit, an unfair detection unit and a notification unit. The action recognition unit recognizes the actions of the shopper relative to the settlement terminal. The operation recognition unit recognizes the operations performed by the shopper on the settlement terminal. The unfair detection unit detects the unfair behavior of the shopper based on the correspondence between the actions of the shopper and the operations performed by the shopper on the settlement terminal. The notification unit notifies the shopper of the unfair behavior.

[0006] According to the above-mentioned fraudulent behavior recognition device, it is possible to provide a fraudulent behavior recognition device that suppresses fraudulent behavior of a customer with respect to a settlement terminal.

[0007] In the above-mentioned fraudulent behavior recognition device, the fraud detection unit detects the fraudulent behavior of the shopper when the order of operations recognized by the operation recognition unit and the actions of the shopper recognized by the action recognition unit are inconsistent.

[0008] According to the above-mentioned fraudulent behavior recognition device, it is possible to detect fraudulent behavior when the operation sequence of the settlement terminal and the behavior of the shopper are inconsistent.

[0009] In the above-mentioned fraudulent behavior recognition device, the fraud detection unit detects the fraudulent behavior of the shopper when the number of operations recognized by the operation recognition unit and the behavior of the shopper recognized by the behavior recognition unit are inconsistent.

[0010] According to the above-mentioned fraudulent behavior recognition device, it is possible to detect fraudulent behavior when the number of operations of the settlement terminal and the actions of the shopper are inconsistent.

[0011] In the above-mentioned fraudulent behavior recognition device, the action recognition unit recognizes the action of the shopper based on the position of the components of the settlement terminal and the position of the hand of the shopper facing the settlement terminal.

[0012] According to the above-mentioned fraudulent behavior recognition device, the behavior of the shopper can be recognized based on the position of the components of the settlement terminal and the position of the shopper's hand.

[0013] In the above-mentioned fraudulent activity recognition device, the operation recognition unit recognizes the operation performed on the settlement terminal by the shopper based on the image converted according to the operation on the settlement terminal.

[0014] According to the above-mentioned fraudulent activity recognition device, it is possible to recognize the operation of the shopper based on the image converted according to the operation of the settlement terminal.

[0015] In the above-mentioned unfair behavior recognition device, the action recognition unit further acquires image data from a camera and stores the image data.

[0016] According to the above-mentioned fraudulent behavior recognition device, when the customer's fraudulent behavior is detected, the stored camera data can be reproduced to verify the fraudulent behavior.

[0017] In the above-mentioned improper behavior identification device, the action identification unit stores first state information representing the identified action, the operation identification unit stores second state information representing the identified operation, and the improper detection unit detects improper behavior based on the situation where the first state information and the second state information are not stored in a prescribed order.

[0018] According to the above-mentioned unfair behavior recognition device, the relationship between actions and movements can be easily determined.

[0019] The storage medium of another aspect of the present invention stores a control program, which is used to enable the computer of the unfair behavior identification device to implement the following functions: a function of identifying a shopper's actions relative to a settlement terminal; a function of identifying operations performed on the settlement terminal by the shopper; a function of detecting the shopper's unfair behavior based on the correspondence between the shopper's actions and the operations performed on the settlement terminal by the shopper; and a function of informing the shopper of the unfair behavior.

[0020] According to the above-mentioned storage medium, it is possible to provide the computer of the fraudulent behavior recognition device with a function of suppressing the customer's fraudulent behavior with respect to the settlement terminal.

[0021] Another aspect of the present invention is a method for identifying unfair behavior, in which an unfair behavior identification device performs the following steps: identifying a shopper's actions relative to a settlement terminal; identifying operations performed on the settlement terminal by the shopper; and detecting the shopper's unfair behavior based on the correspondence between the shopper's actions and the operations performed on the settlement terminal by the shopper.

[0022] According to the above-mentioned fraudulent behavior recognition method, it is possible to provide a fraudulent behavior recognition method that suppresses fraudulent behavior of a customer with respect to a payment terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a system configuration diagram for a store that has introduced a self-service POS terminal.

[0024] Figure 2 It is a diagram for explaining the positional relationship between the self-service POS terminal and the camera;

[0025] Figure 3 is a schematic diagram showing an example of a monitoring image displayed on the display of the waiter terminal;

[0026] Figure 4 This is a block diagram showing the main circuit structure of the improper behavior recognition device;

[0027] Figure 5 is a schematic diagram showing an example of a message list;

[0028] Figure 6 is a schematic diagram showing an example of a timing buffer;

[0029] Figure 7 is a flowchart for explaining the function of the action recognition unit;

[0030] Figure 8 is a flowchart for explaining the function of the operation recognition unit;

[0031] Fig. 9is a flowchart for explaining the function of the fraud detection unit;

[0032] Fig.10 is a flowchart for explaining the function of the fraud detection unit;

[0033] Fig.11 is a flowchart for explaining the function of the fraud detection unit;

[0034] Fig.12 is a flowchart for explaining the function of the notification unit;

[0035] Fig.13 is a flowchart for explaining the function of a fraud detection unit in another embodiment; and

[0036] Fig.14 This is a flowchart for explaining the function of a fraud detection unit in another embodiment.

[0037] Description of Reference Numerals

[0038] 11Self-service POS terminal 12POS server

[0039] 13 display control device 14 waiter terminal

[0040] 15 communication network 21 camera

[0041] 22 Improper behavior recognition device 40 Main body

[0042] 41 touch panel 42 reading window

[0043] 50 bagging stations 81 processors

[0044] 82 Main memory 83 Auxiliary storage device

[0045] 84 clock 85 camera interface

[0046] 86 Communication interface 87 System bus

[0047] 100 Self-service POS system 200 Improper behavior identification system

[0048] 221 action recognition unit 222 operation recognition unit

[0049] 223 Fraud detection unit 224 Notification unit DETAILED DESCRIPTION

[0050] Hereinafter, an embodiment will be described with reference to the drawings.

[0051] Figure 1This is a system configuration diagram of a store that has introduced a self-service POS terminal 11. This system includes a self-service POS system 100 and an improper behavior recognition system 200. The self-service POS system 100 includes a plurality of self-service POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The plurality of self-service POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a LAN (Local Area Network). The LAN can be a wired LAN or a wireless LAN.

[0052] The self-service POS terminal 11 is a fully self-service settlement terminal that allows customers to perform operations from registration of purchased goods to settlement by themselves. Customers are sometimes also called shoppers, consumers, customers, etc. The POS server 12 is a server computer for integrating and controlling the operations of each self-service POS terminal 11. The display control device 13 generates a monitoring image SC (see Figure 1) at each self-service POS terminal 11 based on the data signal output from each self-service POS terminal 11. Figure 3 ) and displays it on the controller of the display device of the waiter terminal 14. The waiter terminal 14 is a terminal used by a store clerk called a waiter to monitor the status of each self-service POS terminal 11. The waiter terminal 14 includes a display such as a liquid crystal display and an organic EL display. The waiter terminal 14 divides the screen of the display into a plurality of areas, and displays a different monitoring image SC of the waiter terminal 14 in each area. Such a self-service POS system 100 can adopt an existing well-known system as it is.

[0053] The fraudulent behavior recognition system 200 includes a plurality of cameras 21 and a fraudulent behavior recognition device 22. The plurality of cameras 21 correspond to the plurality of self-service POS terminals 11, respectively. The cameras 21 are used to photograph the customer who operates the corresponding self-service POS terminal 11. The fraudulent behavior performed by the customer is considered to be the goods to be purchased, that is, the purchase target goods are not registered but bagged, or the high-priced goods among the purchase target goods are repeatedly replaced with low-priced goods and registered, and the total amount is reduced although the number of registered goods is the same as the number of purchased goods.

[0054] The fraudulent behavior recognition device 22 has functions as an action recognition unit 221, an operation recognition unit 222, a fraud detection unit 223, and a notification unit 224. The action recognition unit 221 is a function of recognizing the action of the customer with respect to the self-service POS terminal 11 based on the video data output by each camera 21. The action recognition unit 221 can also be called action recognition means. The operation recognition unit 222 is a function of recognizing the operation of the customer with respect to the self-service POS terminal 11 based on the data of the surveillance image SC output by the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be called operation recognition means. The fraud detection unit 223 is a function of detecting the fraudulent behavior of the customer with respect to the self-service POS terminal 11 based on the recognition result recognized by the action recognition unit 221 and the recognition result recognized by the operation recognition unit 222. The fraud detection unit 223 can also be called fraud detection means. The notification unit 224 is a function of notifying when the fraudulent behavior is detected by the fraud detection unit 223. The notification unit notifies the attendant through the attendant terminal 14, for example. The notification unit 224 can also be called notification means.

[0055] Figure 2 This is a diagram for explaining the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external appearance of the self-service POS terminal 11 will be described.

[0056] The self-service POS terminal 11 includes a main body 40 disposed on the ground and a bagging table 50 disposed next to the main body 40. A touch panel 41 is mounted on the upper portion of the main body 40. The touch panel 41 is composed of a display and a touch sensor. The display is a display for displaying various screens to an operator operating the self-service POS terminal 11. The touch sensor is a device for detecting touch inputs made by the operator to the screen. In the self-service POS terminal 11, the operator is usually a customer.

[0057] The main body 40 is provided with a basket stand 60 at the center of the side opposite to the side where the bagging stand 50 is provided. The basket stand 60 is used to place the baskets etc. of the purchased goods of the customers coming from the store. The customers stand in a manner that they can see the screen of the touch panel 41. Figure 2 Therefore, from the customer's perspective, there is a basket table 60 on the right side and a bagging table 50 on the left side across the main body 40. In the following description, the side where the customer stands is regarded as the front side of the main body 40, the side where the bagging table 50 is provided is regarded as the left side of the main body 40, and the side where the basket table 60 is provided is regarded as the right side of the main body 40.

[0058] The main body 40 has a scanner reading window 42, a card insertion port 43, a ticket dispensing port 44, a coin insertion port 45, a coin dispensing port 46, a bill insertion port 47, and a bill dispensing port 48 formed on its front. In addition, a communication cable 61 extends outward from the right side of the main body 40, and a reader / writer 62 for electronic money media is connected to the front end of the communication cable 61. The reader / writer 62 is set on a placement table 63 set on the upper right side of the main body 40.

[0059] The main body 40 has a display column 64 mounted thereon. The display column 64 includes a light emitting portion 65 at its top end portion that selectively emits blue light and red light. The display column 64 displays the state of the self-service POS terminal 11, such as standby, operation, calling, error, etc., by the light color of the light emitting portion 65.

[0060] The bagging station 50 has a bag holding tool 52 mounted on the upper portion of a housing 51. The bag holding tool 52 has a pair of holding arms 53, and uses the holding arms 53 to hold shopping bags provided in stores or so-called eco-friendly shopping bags carried by customers.

[0061] Next, the positional relationship between the self-service POS terminal 11 and the camera 21 will be described.

[0062] like Figure 2 As shown, the camera 21 is installed in front of the self-service POS terminal 11 and can capture the position of the customer facing the main body 40, the bagging table 50, the basket table 60 and other components from above.

[0063] The customer standing in front of the self-service POS terminal 11 first places a basket containing purchased goods on the basket table 60 on the right side, and holds a shopping bag or an eco-friendly shopping bag on the holding arm 53 on the left side. Then, the customer operates the touch panel 41 according to the guidance displayed on the touch panel 41 and announces the start of the use of the self-service POS terminal 11.

[0064] Then, the customer takes the purchased goods one by one from the basket placed on the basket table 60. Then, when the purchased goods have bar codes attached thereto, the customer registers the goods by aligning the bar codes with the reading window 42 and scanning and reading them. When the purchased goods do not have bar codes attached thereto, the customer registers the goods by operating the touch panel 41 and selecting the purchased goods from the list of goods without bar codes. The customer puts the purchased goods that have been registered into a shopping bag or an eco-friendly shopping bag, etc.

[0065] After registering all purchased goods, the customer operates the touch panel 41 and selects a settlement method. For example, when cash settlement is selected, the customer inserts banknotes or coins into the banknote input port 47 or the coin input port 45, and takes out the change paid out from the banknote dispensing port 48 or the coin dispensing port 46. If, for example, electronic money settlement is selected, the customer aligns the electronic money medium with the reader / writer 62. If, for example, credit card settlement is selected, the customer inserts a credit card into the card insertion port 43. In this way, when settlement is completed, the customer receives the receipt issued from the receipt issuing port 44, and leaves the store with the shopping bag or eco-friendly shopping bag removed from the holding arm 53.

[0066] That is, the camera 21 is installed at a position on the front of the self-POS terminal 11 so as to be able to capture the movement of the hand of the customer who is performing the above-mentioned actions.

[0067] Figure 3 1 is a schematic diagram showing an example of a monitoring screen SC displayed on the display of the attendant terminal 14. As described above, the monitoring screen SC is displayed on the display of the attendant terminal 14 in a partitioned manner corresponding to each of the plurality of self-service POS terminals 11. Figure 3 This is an example of the monitoring image SC corresponding to one of the self-service POS terminals 11. Since the configuration of the monitoring images SC corresponding to the other self-service POS terminals 11 is the same as this, the description thereof will be omitted here.

[0068] like Figure 3 As shown, the monitoring image SC includes a cash register number column 71 , a terminal status column 72 , an error information column 73 , a notification information column 74 , a detail column 75 , and a total column 76 .

[0069] The cash register number field 71 is a field for displaying the cash register number. The cash register number is a series of numbers assigned to each self-service POS terminal 11 so as to not overlap in order to identify each self-service POS terminal 11. The cash register number is identification information for identifying each self-service POS terminal 11.

[0070] The terminal status column 72 is a column for displaying the operating status of the self-service POS terminal 11. In this embodiment, the terminal status column 72 displays any one of "waiting", "starting use", "registering", "starting settlement" and "settling settlement" as the operating status.

[0071] "Standby" is a state that is announced after the previous customer has completed settlement and the next customer starts using the product. The touch panel of the self-service POS terminal 11 in the "Standby" state displays an initial image. The initial image includes, for example, an image containing a touch button for the customer to choose to use a shopping bag provided by the store or to use an eco-friendly shopping bag.

[0072] "Use start" is a state where the customer standing in front of the self-service POS terminal 11 declares the start of use for settlement. The customer performs a selection operation for the initial image to use a shopping bag or to use an eco-friendly shopping bag. The selection operation is declared to start use. After receiving the selection operation, the operation state of the self-service POS terminal 11 changes to "use start".

[0073] "Registering" is a state in which the registration operation of the purchased goods by the customer himself is accepted. When the first purchased goods are registered, the operation state of the self-service POS terminal 11 changes to "Registering". Thereafter, until the transition to settlement is announced, the operation state of the self-service POS terminal 11 remains in "Registering".

[0074] "Settlement Start" is a state in which the customer who has completed the registration of purchasing goods declares the transition to settlement. The touch panel 41 of the self-service POS terminal 11 in the "Registration" state displays a "Checkout" soft key. The customer who has completed the registration of purchasing goods touches the "Checkout" soft key. This operation declares the transition to settlement. After accepting this operation, the action state of the self-service POS terminal 11 changes to "Settlement Start".

[0075] "Settlement in progress" is a state in which settlement processing such as cash settlement, electronic money settlement, credit card settlement, etc. is being executed. For example, when banknotes or coins are inserted into the banknote insertion port 47 or the coin insertion port 45, the operation state of the self-service POS terminal 11 is "Settlement in progress". Then, when the settlement processing is completed, the operation state of the self-service POS terminal 11 returns to "Standby".

[0076] The error information column 73 is a column for displaying error information that occurs in the self-service POS terminal 11. The error information is a communication error, a ticket out error, etc. The announcement information column 74 is a column for displaying the content of the customer's announcement operation. For example, when the customer selects an environmentally friendly shopping bag, "No shopping bag" is displayed to indicate that the shopping bag is not needed.

[0077] The detail column 75 is a column for displaying the detail information of the purchased goods registered using the self-service POS terminal 11. The detail information includes, for example, the name of the purchased goods, the number of items, the amount, etc. The total column 76 is a column for displaying the total information of the purchased goods registered using the self-service POS terminal 11. The total information includes the total number of items, the total amount, the deposited amount, the change, etc.

[0078] In addition, the configuration of the monitoring image SC is not limited to this. A column displaying other items may also be configured. Figure 3 The items of the text data in the display may not be limited to these. The text data of other items may also be displayed.

[0079] Figure 48 is a block diagram showing the main circuit structure of the fraudulent behavior recognition device 22. The fraudulent behavior recognition device 22 includes a processor 81, a main memory 82, an auxiliary storage device 83, a clock 84, a camera interface 85, a communication interface 86, and a system bus 87. The system bus 87 includes an address bus and a data bus, etc. The fraudulent behavior recognition device 22 is configured as a computer by connecting the processor 81, the main memory 82, the auxiliary storage device 83, the clock 84, the camera interface 85, and the communication interface 86 using the system bus 87.

[0080] The processor 81 corresponds to the core part of the above-mentioned computer. The processor 81 controls each part to realize various functions of the fraudulent behavior recognition device 22 according to the operating system or application program. The processor 81 is, for example, a CPU (Central Processing Unit).

[0081] The main memory 82 is equivalent to the main storage part of the above-mentioned computer. The main memory 82 includes a non-volatile memory area and a volatile memory area. The main memory 82 stores an operating system or an application program in the non-volatile memory area. The main memory 82 stores necessary data for the processing executed by the processor 81 for controlling each part in the volatile memory area. Such data is sometimes also stored in the non-volatile memory area. The main memory 82 uses the volatile memory area as a working area where data can be appropriately rewritten by the processor 81. An example of a non-volatile memory area is a ROM (Read Only Memory). An example of a volatile memory area is a RAM (Random Access Memory).

[0082] The auxiliary storage device 83 is equivalent to the auxiliary storage part of the above-mentioned computer. As the auxiliary storage device 83, for example, a well-known storage device such as an SSD (Solid State Drive), an HDD (Hard Disc Drive) or an EEPROM (Electric Erasable Programmable Read-Only Memory) is used alone or in combination of multiple devices. The auxiliary storage device 83 stores data used in various processing processes performed by the processor 81, data generated based on the processing of the processor 81, etc. The auxiliary storage device 83 sometimes also stores application programs. It can also be used as a storage unit for storing camera data acquired from each camera 21.

[0083] The application programs stored in the main memory 82 or the auxiliary storage device 83 include a control program described later. The method of installing the control program in the main memory 82 or the auxiliary storage device 83 is not particularly limited. The control program can be stored in a removable storage medium, or the control program can be transmitted by communication via a network and installed in the main memory 82 or the auxiliary storage device 83. The storage medium can store the program in the form of a CD-ROM, a memory card, etc., and its form is not limited as long as the device can read it.

[0084] The clock 84 functions as a time information source of the fraudulent activity recognition device 22. The processor 81 acquires the current date and time based on the time information measured by the clock 84.

[0085] The camera interface 85 is an interface for communicating with each camera 21. The image data output from each camera 21 is acquired by the fraudulent behavior recognition device 22 through the camera interface 85. The image data is data of a customer operating the self-service POS terminal 11 corresponding to the camera 21.

[0086] The communication interface 86 is an interface for communicating with the display control device 13. The image data output from the display control device 13 is acquired by the fraudulent activity recognition device 22 through the communication interface 86. The image data is data of the surveillance image SC generated by each self-service POS terminal 11.

[0087] The fraudulent behavior recognition device 22 of the related structure uses a part of the volatile memory area in the main memory 82 as an area for the message list 821. Then, the fraudulent behavior recognition device 22 forms a message list 821 in the area. Figure 5 The data structure shown is a message list 821.

[0088] Figure 5 821 is a schematic diagram showing an example of a message list 821. Figure 5 As shown, the message list 821 is a data list in which text data of messages are recorded in association with the error code EC. The error code EC and the text data of the message associated therewith will be explained in detail in the following description.

[0089] In addition, the fraudulent behavior recognition device 22 uses a part of the volatile memory area in the main memory 82 as an area of ​​the timing buffer 822. Then, the fraudulent behavior recognition device 22 forms only the number of self-service POS terminals 11 in this area. Figure 6 The data structure of the timing buffer 822 is shown.

[0090] Figure 6 822 is a schematic diagram showing an example of the timing buffer 822. Figure 6As shown, the timing buffer 822 has an area for recording the cash register number identifying the self-service POS terminal 11 and an area for recording the status ST or error code EC in chronological order of time TM. The status ST will be explained in detail later.

[0091] Therefore, in order to suppress customers' improper behavior towards the self-service POS terminal 11, the improper behavior identification device 22 realizes the functions of the action identification unit 221, operation identification unit 222, improper detection unit 223 and notification unit 224 through the processor 81 and the control program that controls the processor 81.

[0092] The functions of the action recognition unit 221, the operation recognition unit 222, the fraud detection unit 223, and the notification unit 224 are functions that each self-service POS terminal 11 has. Therefore, the functions of the action recognition unit 221, the operation recognition unit 222, the fraud detection unit 223, and the notification unit 224 corresponding to one self-service POS terminal 11 will be described in detail below. The functions of the action recognition unit 221, the operation recognition unit 222, the fraud detection unit 223, and the notification unit 224 corresponding to other self-service POS terminals 11 are the same, and the description thereof is omitted here.

[0093] Figure 7 This is a flowchart for explaining the function of the action recognition unit 221.

[0094] As ACT 1, the processor 81 waits for the reception of the customer recognition. The camera 21 is set at a position where it can capture the customer standing in front of the self-service POS terminal 11 from above. Therefore, when the processor 81 detects that a person is standing in front of the self-service POS terminal 11 from the image data of the camera 21, it determines that the customer has been recognized.

[0095] After the customer is recognized, the processor 81 determines YES in ACT 1 and moves to ACT 2. As ACT 2, the processor 81 obtains the cash register number of the self-service POS terminal 11. Each camera 21 corresponds to each self-service POS terminal 11. Therefore, the processor 81 specifies the self-service POS terminal 11 from the identification information of the camera 21 that photographs the customer standing in front of the self-service POS terminal 11, and obtains the cash register number of the self-service POS terminal 11.

[0096] As ACT 3, the processor 81 starts the guest's action recognition. Specifically, the processor 81 estimates the skeleton of the person, that is, the guest, from the image captured by the camera 21. Skeleton estimation can be realized even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the guest's taking-out action, registering action, bagging action, or leaving action from the hand movement obtained by skeleton estimation.

[0097] The taking-out action is an action of taking out the purchased goods from the basket placed on the basket stand 60. For example, when a movement such as the skeleton of one or both hands moving on the right side of the main body 40 to pick up the purchased goods is detected, the processor 81 recognizes that there is a taking-out action.

[0098] The registration action is an action of registering the data of the purchased goods taken out of the basket in the self-service POS terminal 11. For example, when the action of the skeleton of the hand that performs the taking action aligning the purchased goods with the reading window 42 in the center of the main body 40 is detected, the processor 81 recognizes that there is a registration action. Alternatively, when the action of the skeleton of a single hand operating the touch panel 41 of the main body 40 is detected, the processor 81 recognizes that there is a registration action.

[0099] The bagging action is an action of placing the registered purchased goods into a shopping bag or an eco-friendly shopping bag on the bagging station 50. For example, when the hand skeleton that performed the registration action moves to the left side of the main body and moves the purchased goods into the shopping bag or the eco-friendly shopping bag, the processor 81 recognizes that there is a bagging action.

[0100] The check-out action is the action of the customer who has completed the settlement to leave the self-service POS terminal 11. For example, when the hand skeleton of the customer who has completed the settlement makes an action of removing a shopping bag or a recyclable shopping bag from the holding arm 53, if the camera data of the camera 21 cannot detect the customer, the processor 81 recognizes that there is a check-out action.

[0101] As ACT4 to ACT7, the processor 81 waits for receiving the identification of the take-out action, the registration action, the bagging action or the check-out action. In this waiting state, when the take-out action is identified, the processor 81 determines YES in ACT4 and proceeds to ACT8. As ACT8, the processor 81 makes the state ST "11". If the registration action is identified, the processor 81 determines YES in ACT5 and proceeds to ACT9. As ACT9, the processor 81 makes the state ST "12". When the bagging action is identified, the processor 81 determines YES in ACT6 and proceeds to ACT10. As ACT10, the processor 81 makes the state ST "13". "11", "12", "13" of the state ST are state information representing the identified action. According to the action identified by the action identification unit 221, the state information stored in the main memory 82 is made the first state information.

[0102] After completing the processing of ACT8, ACT9 or ACT10, the processor 81 proceeds to ACT11. As ACT11, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT12, the processor 81 records the time TM and the state ST in association with each other in the timing buffer 822 set with the cash register number obtained by the processing of ACT2. Then, the processor 81 returns to the waiting state of ACT4 to ACT7.

[0103] Thus, when the action of taking out the self-service POS terminal 11 is recognized by the function of the action recognition unit 221, the state ST is recorded as "11" together with the time TM in the timing buffer 822 set with the cash register number of the self-service POS terminal 11. Similarly, when the action of registering is recognized, the state ST is recorded as "12" together with the time TM in the same timing buffer 822. When the action of bagging is recognized, the state ST is recorded as "13" together with the time TM in the same timing buffer 822.

[0104] Usually, the customer registers the data of the purchased goods at the self-service POS terminal 11 by repeating the taking-out action, registering action, and bagging action in sequence at the self-service POS terminal 11. Therefore, in the timing buffer 822, the states ST are stored in the order of "11", "12", and "13" in a timing sequence.

[0105] When the check-out action is recognized in the waiting state from ACT 4 to ACT 7, the processor 81 determines YES in ACT 7 and proceeds to ACT 13. As ACT 13, the processor 81 ends the recognition of the guest's action.

[0106] The processor 81, which has completed the action recognition, sets the state ST to "14" as ACT14. In addition, as ACT15, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT16, the processor 81 records the time TM and the state ST in association in the timing buffer 822 set with the cash register number obtained by the processing of ACT2. Therefore, in the timing buffer 822 corresponding to the self-service POS terminal 11 where the customer leaves, the state ST is stored as "14" together with the time TM.

[0107] At this point, the processor 81 ends the function as the action recognition unit 221. Then, when the image data of the camera 21 detects a person standing in front of the self-service POS terminal 11 again, the processor 81 executes the processes of ACT 2 to ACT 16 in the same manner as described above.

[0108] Figure 8 This is a flowchart for explaining the function of the operation recognition unit 222 .

[0109] As ACT 21, the processor 81 waits for receiving a declaration of use start for the self-service POS terminal 11. When the use start is declared, "use start" is displayed in the terminal status column 72 of the monitoring image SC corresponding to the self-service POS terminal 11. The processor 81 confirms whether the characters "use start" can be recognized in the terminal status column 72 of the monitoring image SC obtained by the display control device 13. When the characters "use start" can be recognized, the processor 81 recognizes that the use start is declared.

[0110] When the processor 81 recognizes that the use start is declared, it determines YES in ACT 1 and proceeds to ACT 22. As ACT 22, the processor 81 obtains the cash register number of the self-service POS terminal 11. The cash register number is displayed in the cash register number column 71 of the monitoring image SC. The processor 81 recognizes the characters of the cash register number from the cash register number column 71 of the monitoring image SC obtained by the display control device 13, and obtains the characters as the cash register number.

[0111] As ACT 23, the processor 81 sets the state ST to "21" indicating the use start operation. In addition, as ACT 24, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 25, the processor 81 records the time TM and the state ST in association with each other in the timing buffer 822 set with the cash register number obtained by the processing of ACT 22.

[0112] Therefore, when the customer standing in front of the self-service POS terminal 11 performs a declaration operation for starting use, the state ST "21" is first stored in the timing buffer 822 corresponding to the self-service POS terminal 11 together with the time TM.

[0113] As ACT 26, the processor 81 starts to recognize an operation on the self-service POS terminal 11. Specifically, the processor 81 converts information obtained by character recognition of the surveillance image SC acquired by the display control device 13 to recognize a commodity registration operation, a registration deletion operation, a settlement start operation, or a settlement end operation.

[0114] For example, when the processor 81 adds detailed information such as the product name, number of pieces, and amount of the purchased product to the detail column 75, it recognizes that there is a product registration operation. For example, when the total number of pieces or the total amount displayed in the total column 76 decreases, the processor 81 recognizes that there is a registration deletion operation. For example, when the display of the terminal status column 72 switches to "settlement start", the processor 81 recognizes that there is a settlement start operation. For example, when the display of the terminal status column 72 switches to "standby", the processor 81 recognizes that there is a settlement end operation.

[0115] As ACT27 to ACT30, the processor 81 waits for receiving the identification of the product registration operation, the registration deletion operation, the settlement start operation or the settlement end operation. When the product registration operation is recognized, the processor 81 determines YES in ACT27 and proceeds to ACT31. As ACT31, the processor 81 makes the state ST "22". When the registration deletion operation is recognized, the processor 81 determines YES in ACT28 and proceeds to ACT32. As ACT32, the processor 81 makes the state ST "23". When the settlement start operation is recognized, the processor 81 determines YES in ACT29 and proceeds to ACT33. As ACT33, the processor 81 makes the state ST "24". "21", "22" and "23" of the state ST are status information indicating the recognized operation. According to the operation recognized by the operation recognition unit 222, the status information stored in the main memory 82 is the second status information.

[0116] After completing the processing of ACT 31, ACT 32, or ACT 33, the processor 81 proceeds to ACT 34. As ACT 34, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 35, the processor 81 records the time TM and the state ST in association with each other in the timing buffer 822 set with the cash register number obtained by the processing of ACT 22. Then, the processor 81 returns to the waiting state of ACT 27 to ACT 30.

[0117] Thus, when the operation recognition unit 222 recognizes a commodity registration operation for the self-service POS terminal 11, the state ST is recorded as "22" together with the time TM in the timing buffer 822 set with the cash register number of the self-service POS terminal 11. Similarly, when a registration deletion operation is recognized, the state ST is recorded as "23" together with the time TM in the same timing buffer 822. When a settlement start operation is recognized, the state ST is recorded as "24" together with the time TM in the same timing buffer 822.

[0118] Usually, the customer's registration action performs a commodity registration operation or a registration deletion operation on the self-service POS terminal 11. Therefore, in the timing buffer 822, after the state ST indicating the registration action, i.e., "12", the state indicating the commodity registration operation or the registration deletion operation, i.e., "22" or "23", is stored.

[0119] In addition, the customer who has finished the bagging operation of all purchased commodities announces the start of settlement. Therefore, in the timing buffer 822, after the state ST representing the bagging operation, i.e., "13", the state representing the start of settlement operation, i.e., "24", is stored.

[0120] In the waiting state from ACT 27 to ACT 30, when the processor 81 recognizes the settlement completion operation, the processor 81 determines YES in ACT 30 and proceeds to ACT 36. In ACT 36, the processor 81 ends the recognition of the operation to the self-service POS terminal 11.

[0121] As ACT 37, the processor 81 that has completed the operation recognition sets the state ST to "25". In addition, as ACT 38, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 39, the processor 81 records the time TM and the state ST = 25 in association with each other in the timing buffer 822 set with the cash register number obtained by the processing of ACT 22. Therefore, in the timing buffer 822 corresponding to the self-service POS terminal 11 that has completed the settlement, the state ST "25" is stored together with the time TM.

[0122] At this point, the processor 81 ends the function as the operation recognition unit 222. Thereafter, when the use start operation of the self-service POS terminal 11 is detected again from the data on the monitoring screen SC, the processor 81 executes the processes of ACT 22 to ACT 29 in the same manner as described above.

[0123] Figures 9 to 11 This is a flowchart for explaining the function of the fraud detection unit 223.

[0124] As ACT 41, the processor 81 waits for the action recognition unit 221 to recognize the withdrawal action. As described above, the timing buffer 822 in which the cash register number of the self-service POS terminal 11 that recognized the withdrawal action is set has the state ST recorded as "11". When the processor 81 detects that the state ST is recorded as "11" in the timing buffer 822, it determines YES in ACT 41 and proceeds to ACT 42.

[0125] As ACT 42, the processor 81 confirms whether it is after the use start operation. When the use start operation is performed on the self-service POS terminal 11, the state ST is recorded as "21" in the timing buffer 822. Therefore, the processor 81 searches the timing buffer 822 in the back-tracking time direction from the moment TM when the state ST is recorded as "11". Then, when it is detected that the state is recorded as "21", the processor 81 determines that it is after the use start operation. Correspondingly, when the state is detected as "14" or "25", or the search of the timing buffer 822 is completed, the processor 81 determines that it is not after the use start operation. When it is detected that the state information is not stored in the prescribed order, improper behavior is detected. Then, the error code EC indicating the improper behavior is stored.

[0126] If it is not after the start of use, the processor 81 determines NO in ACT 42 and proceeds to ACT 43. As ACT 43, the processor 81 sets the error code EC to "91". In addition, as ACT 44, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 45, the processor 81 records the time TM and the error code EC in the timing buffer 822 in association. Thereafter, the processor 81 proceeds to ACT 46.

[0127] On the other hand, if the use has started, the processor 81 determines YES in ACT 42. The processor 81 skips the processes of ACT 43 to ACT 45 and proceeds to ACT 46.

[0128] Therefore, in the self-service POS terminal 11 that has not performed the use start announcement operation, when the removal action of the commodity is recognized, the error code EC "91" is stored in the sequence buffer 822 set with the cash register number of the self-service POS terminal 11.

[0129] Thus, although the use start operation is not recognized, when the withdrawal action is recognized, the error code EC is recorded as "91" in the timing buffer 822. That is, the error code EC "91" is a code that recognizes the behavior of the guest who performs the withdrawal action without performing the use start declaration operation as an improper behavior "improper withdrawal".

[0130] As ACT 46, the processor 81 checks whether the bagging action is recognized by the action recognition unit 221. As described above, when the bagging action is recognized, the state ST is recorded as "12" in the timing buffer 822. When the state ST is not recorded as "12" in the timing buffer 822, the processor 81 determines NO in ACT 46 and proceeds to ACT 47.

[0131] As ACT 47, the processor 81 checks whether the fetching action is recognized by the action recognition unit 221. As described above, when the fetching action is recognized, the state ST is recorded as "11" in the timing buffer 822. When the state ST is not recorded as "11" in the timing buffer 822, the processor 81 determines NO in ACT 47 and proceeds to 48.

[0132] As ACT 48, the processor 81 checks whether the store exit action is recognized by the action recognition unit 221. As described above, when the store exit action is recognized, the status ST is recorded as "14" in the timing buffer 822. If the status is not recorded as "14" in the timing buffer 822, the processor 81 determines NO in ACT 48 and returns to ACT 46.

[0133] Here, the processor 81 waits in ACT 46 to ACT 48 for the customer who has taken out the first product to pack the product, or to obtain other products, or to leave the store.

[0134] In this waiting state, when the state ST recorded in the timing buffer 822 is "13", the processor 81 determines YES in ACT 46 and proceeds to Fig.10 ACT61.

[0135] As ACT 61, the processor 81 checks whether the operation is after the product registration operation. As described above, when the product registration operation is recognized in the operation recognition unit 222, the state ST is recorded as "22" in the timing buffer 822. Therefore, when the state ST is recorded as "22" in association with the time TM before the time TM at which the state ST is recorded as "13", the processor 81 recognizes that the product registration operation is after. The processor 81 determines NO in ACT 61 and returns to Fig. 9 That is, the processor 81 returns to the waiting state for receiving ACT46 to ACT48.

[0136] In contrast, when the state ST is not recorded as "22" in association with the time TM before the time TM at which the state ST is recorded as "13", the processor 81 recognizes that the product registration operation is not performed. The processor 81 determines YES in ACT 61 and proceeds to ACT 62.

[0137] As ACT 62, the processor 81 confirms whether it is after the registration and deletion operation. As described above, when the registration and deletion operation is recognized in the operation recognition unit 222, the state ST is recorded as "23" in the timing buffer 822. Therefore, when the state ST is recorded as "23" in association with the moment TM before the moment TM when the state ST is recorded as "13", the processor 81 recognizes that it is after the registration and deletion operation. In other words, when the state ST is recorded as "23" without being associated with the moment TM before the moment TM when the state ST is recorded as "13", the processor 81 recognizes that it is not after the registration and deletion operation.

[0138] If the deletion operation is not performed after the registration, the processor 81 determines NO in ACT 62 and proceeds to ACT 63. As ACT 63, the processor 81 sets the error code EC to "92". In addition, as ACT 64, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 65, the processor 81 associates the time TM with the error code EC in the timing buffer 822 whose recording state ST is "13" and records them. After that, the processor 81 returns to Fig. 9That is, the processor 81 returns to the waiting state for receiving ACT46 to ACT48.

[0139] Thus, although the commodity registration operation or registration deletion operation is not recognized, when the bagging action is recognized, the error code EC "92" is stored in the timing buffer 822. That is, the error code EC "92" is a code that recognizes the customer's behavior of bagging commodities without commodity registration operation as an improper behavior "false registration".

[0140] On the other hand, when the deletion operation is registered, the processor 81 determines YES in ACT 62 and proceeds to ACT 66. As ACT 66, the processor 81 records the error code EC as "93". In addition, as ACT 67, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 68, the processor 81 records the time TM and the error code EC in association in the timing buffer 822 where the state ST is recorded as "13". After that, the processor 81 returns to Fig. 9 That is, the processor 81 returns to the waiting state for receiving ACT46 to ACT48.

[0141] In this way, when the bagging action is recognized after the registration deletion operation is recognized, the error code EC is recorded as "93" in the timing buffer 822. That is, the error code EC "93" is a code that recognizes the behavior of the customer who has bagged the goods that have been registered and deleted as "false cancellation".

[0142] Return to Fig. 9 Description.

[0143] In the waiting state of ACT46 to ACT48, when the state ST is recorded as "11" in the timing buffer 822 and the fetch action is recognized, the processor 81 proceeds to Fig.11 ACT71. As ACT71, the processor 81 confirms whether it is after the settlement start operation. As described above, after the settlement start operation is identified in the operation identification unit 222, the state ST is recorded as "24" in the timing buffer 822. Therefore, when the state ST is recorded as "24" in association with the moment TM before the moment TM when the state ST is recorded as "11", the processor 81 recognizes that it is after the settlement start operation. In other words, when the state ST is not recorded as "24" in association with the moment TM before the moment TM when the state ST is recorded as "11", the processor 81 recognizes that it is not after the settlement start operation.

[0144] When it is not after the settlement start operation, the processor 81 determines NO in ACT71 and proceeds to ACT72. As ACT72, the processor 81 confirms whether it is after the settlement end operation. As described above, when it is identified in the operation identification unit 222 that it is after the settlement end operation, the state ST is recorded as "25" in the timing buffer 822. Therefore, when the state ST is recorded as "25" in association with the moment TM before the moment TM when the state ST is recorded as "11", the processor 81 recognizes that it is after the settlement end operation. In other words, when the state ST is not recorded as "25" in association with the moment TM before the moment TM when the state ST is recorded as "11", the processor 81 recognizes that it is not after the settlement end operation.

[0145] If the settlement operation is not completed, the processor 81 determines that ACT 72 is NO and returns to ACT 72. Fig. 9 That is, the processor 81 returns to the waiting state for receiving ACT46 to ACT48.

[0146] On the other hand, when settlement starts or ends, processor 81 determines YES in ACT 71 or ACT 72 and proceeds to ACT 73. As ACT 73, processor 81 sets error code EC to "94". In addition, as ACT 74, processor 81 obtains the current time TM measured by clock 84. Then, as ACT 75, processor 81 records the time TM and error code EC in association in timing buffer 822 in which state ST is recorded as "11". After that, processor 81 returns to Fig. 9 That is, the processor 81 returns to the waiting state for receiving ACT46 to ACT48.

[0147] Thus, although it is recognized as the settlement start operation or settlement end operation, when the commodity removal action is recognized, the error code EC "94" is stored in the timing buffer 822. That is, the error code EC "94" identifies the customer's behavior of taking out unregistered commodities from the basket after the settlement start operation or settlement end operation as an improper behavior of "omission of registration".

[0148] return Fig. 9 Description.

[0149] In the waiting acceptance state of ACT46 to ACT48, when the status ST is recorded as "14" in the timing buffer 822 and the check-out action is recognized, the processor 81 determines YES in ACT48 and proceeds to ACT49. As ACT49, the processor 81 confirms whether it is after the settlement completion operation. As described above, after the settlement completion operation is recognized in the operation recognition unit 222, the status ST is recorded as "25" in the timing buffer 822. Therefore, when the status ST is recorded as "25" in association with the moment before the moment TM when the status ST is recorded as "14", the processor 81 recognizes that it is after the settlement completion operation. In other words, when the status ST is not recorded as "25" in association with the moment TM before the moment TM when the status ST is recorded as "14", the processor 81 recognizes that it is not after the settlement completion operation.

[0150] If the settlement operation is completed, the processor 81 determines YES in ACT 49 , and the processor 81 terminates the function as the fraud detection unit 223 .

[0151] Correspondingly, when the settlement end operation is not performed, the processor 81 determines NO in ACT 49 and proceeds to ACT 50. As ACT 50, the processor 81 sets the error code EC to "95". In addition, as ACT 51, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT 52, the processor 81 records the time TM and the error code EC in association in the timing buffer 822 in which the state ST is recorded as "14".

[0152] Thus, when the checkout action is recognized even though the settlement completion operation is not recognized, the error code EC is "95" stored in the timing buffer 822. That is, the error code EC "95" is a code that recognizes the behavior of the guest who has checked out before the settlement completion operation as an improper behavior "non-settlement".

[0153] At this point, the processor 81 ends the function as the fraud detection unit 223. Thereafter, when the taking-out action is recognized again, the processor 81 executes the processes of ACT 42 to ACT 52, ACT 61 to ACT 68, and ACT 71 to ACT 75 in the same manner as described above.

[0154] Fig.12 This is a flowchart for explaining the function of the notification unit 224 .

[0155] As ACT 81, the processor 81 waits for the error code EC to be recorded in the sequence buffer 822. When the error code EC is recorded in the sequence buffer 822, the processor 81 determines YES in ACT 81 and proceeds to ACT 82. As ACT 82, the processor 81 confirms whether the error code EC is "91".

[0156] When the error code EC is "91", the processor 81 determines YES in ACT82 and proceeds to ACT83. As ACT83, the processor 81 notifies the improper behavior "illegal withdrawal". That is, the processor 81 searches the message list 821 and obtains the message data of the error code EC "91". In addition, the processor 81 obtains the cash register number of the timing buffer 822 in which the error code EC is recorded as "91". Then, the processor 81 outputs the notification command to which the message data and the cash register number are attached from the communication interface 86 to the display control device 13.

[0157] The display control device 13 displays the text of the message data in the monitoring image SC of the waiter terminal 14 identified by the cash register number included in the notification command. As a result, a text notifying the improper behavior, for example, "an improper withdrawal was made at the cash register No. X" is displayed in the monitoring image SC. Therefore, the waiter can be alerted to the improper behavior "illegal withdrawal" of the customer using the self-service POS terminal 11 set as the cash register No. X.

[0158] When the error code EC is not "91", the processor 81 determines NO in ACT 82 and proceeds to ACT 84. As ACT 84, the processor 81 confirms whether the error code is "92".

[0159] When the error code EC is "92", the processor 81 determines YES in ACT84 and proceeds to ACT85. As ACT85, the processor 81 notifies the unfair behavior "false registration". That is, the processor 81 searches the message list 821 and obtains the message data of the error code EC "92". In addition, the processor 81 obtains the cash register number of the timing buffer 822 in which the error code EC is recorded as "92". Then, the processor 81 outputs the notification command to which the message data and the cash register number are attached from the communication interface 86 to the display control device 13.

[0160] The display control device 13 displays the text of the message data in the monitoring image SC of the waiter terminal 14 identified by the cash register number included in the notification command. As a result, a text notifying that, for example, "false registration was made at the cash register No. X" is an improper behavior is displayed in the monitoring image SC. Therefore, the waiter can be alerted to the improper behavior of "false registration" by the customer using the self-service POS terminal 11 set as the cash register No. X.

[0161] When the error code EC is not "92", the processor 81 determines NO in ACT 84 and proceeds to ACT 86. As ACT 86, the processor 81 confirms whether the error code is "93".

[0162] When the error code EC is "93", the processor 81 determines YES in ACT86 and proceeds to ACT87. As ACT87, the processor 81 performs notification of the unfair behavior "false cancellation". That is, the processor 81 searches the message list 821 and obtains the message data with the error code EC being "93". In addition, the processor 81 obtains the cash register number of the timing buffer 822 in which the error code EC is recorded as "93". Then, the processor 81 outputs the notification command with the message data and the cash register number attached from the communication interface 86 to the display control device 13.

[0163] The display control device 13 displays the text of the message data in the monitoring image SC of the waiter terminal 14 identified by the cash register number included in the notification command. As a result, a text notifying, for example, that an unfair behavior "false cancellation was performed at the cash register No. X" is displayed in the monitoring image SC. Therefore, the waiter can be alerted to the unfair behavior of "false cancellation" by the customer using the self-service POS terminal 11 set as the cash register No. X.

[0164] When the error code EC is not "93", the processor 81 determines "NO" in ACT 86 and proceeds to ACT 88. As ACT 88, the processor 81 confirms whether the error code EC is "94".

[0165] When the error code EC is "94", the processor 81 determines YES in ACT88 and proceeds to ACT89. As ACT89, the processor 81 performs notification of the unfair behavior "omission of registration". That is, the processor 81 searches the message list 821 and obtains the message data with the error code EC being "94". In addition, the processor 81 obtains the cash register number of the timing buffer 822 in which the error code EC is recorded as "94". Then, the processor 81 outputs the notification command with the message data and the cash register number attached from the communication interface 86 to the display control device 13.

[0166] The display control device 13 displays the text of the message data in the monitoring image SC of the waiter terminal 14 identified by the cash register number included in the notification command. As a result, a text notifying that, for example, "there is a misconduct of missing registration at the cash register No. X" is displayed in the monitoring image SC. Therefore, the waiter can be alerted to the misconduct of "missing registration" by the customer using the self-service POS terminal 11 set as the cash register No. X.

[0167] When the error code EC is not "94", the processor 81 determines NO in ACT 88 and proceeds to ACT 90. As ACT 90, the processor 81 confirms whether the error code EC is "95".

[0168] When the error code EC is "95", the processor 81 determines YES in ACT90 and proceeds to ACT91. As ACT91, the processor 81 performs notification of the unfair behavior of "non-settlement". That is, the processor 81 searches the message list 821 and obtains the message data with the error code EC "95". In addition, the processor 81 obtains the cash register number of the timing buffer 822 in which the error code EC "95" is recorded. Then, the processor 81 outputs the notification command of the attached message data and the cash register number from the communication interface 86 to the display control device 13.

[0169] The display control device 13 displays the text of the message data in the monitoring image SC of the waiter terminal 14 identified by the cash register number included in the notification command. As a result, a text notifying the fraudulent behavior, such as "the cash register of cash register No. X has not been settled" is displayed in the monitoring image SC. Therefore, the waiter can be alerted to the fraudulent behavior of "not settling" of the customer using the self-service POS terminal 11 set as cash register No. X.

[0170] As described in detail above, the fraudulent behavior recognition device 22 has a function of recognizing the customer's behavior on the self-service POS terminal 11 based on the video data of the camera 21 as the action recognition unit 221. In addition, the fraudulent behavior recognition device 22 has a function of recognizing the operation performed by the customer on the self-service POS terminal 11 based on the conversion of the surveillance image SC displayed on the waiter terminal 14 as the operation recognition unit 222. Furthermore, the fraudulent behavior recognition device 22 has a function of detecting the customer's fraudulent behavior based on the correspondence between the customer's behavior and the operation performed by the customer on the self-service POS terminal 11 as the fraud detection unit 223. That is, when the order of the operation recognized by the fraudulent behavior recognition device 22 through the operation recognition unit 222 contradicts the customer's behavior recognized by the action recognition unit 221, the customer's fraudulent behavior is detected.

[0171] For example, when the withdrawal action is recognized without the use of the start declaration operation, the unfair behavior "illegal withdrawal" is detected. When the bagging action of the goods is recognized without the goods registration operation, the unfair behavior "false registration" is detected. When the bagging action of the goods is recognized after the registration deletion operation, the unfair behavior "false cancellation" is detected. When the withdrawal action is recognized after the settlement start operation or the settlement end operation, the unfair behavior "omission of registration" is detected. When the store checkout action is recognized without the settlement end operation, the unfair behavior "non-settlement" is detected.

[0172] The fraudulent behavior recognition device 22 as the notification unit 224 has the function of notifying the waiter of the customer's fraudulent behavior using the waiter terminal 14. Therefore, the waiter who has confirmed the customer's fraudulent behavior can quickly notice the customer. As a result, the customer's fraudulent behavior with respect to the self-service POS terminal 11 can be suppressed.

[0173] In this way, according to this embodiment, it is possible to achieve the effect of suppressing various improper behaviors of guests.

[0174] An embodiment of the technology for suppressing unfair behavior of customers with respect to the self-service POS terminal 11 has been described above, but the embodiment is not limited thereto.

[0175] In the above embodiment, the case where the fraud detection unit 223 detects the guest's fraudulent behavior when the order of operations recognized by the operation recognition unit 222 contradicts the guest's behavior recognized by the action recognition unit 221 is taken as an example. The detection technology of the fraudulent behavior by the fraud detection unit 223 is not limited to this. For example, consider that when the number of operations recognized by the operation recognition unit 222 contradicts the guest's behavior recognized by the action recognition unit 221, the fraud detection unit 223 detects the guest's fraudulent behavior.

[0176] Fig.13 and Fig.14 Each of them is a flowchart for explaining the function of the fraud detection unit 223 for detecting the fraudulent behavior "false registration" based on the number of operations and the behavior of the customer.

[0177] exist Fig.13In ACT 101, the processor 81 waits for the settlement start operation to be recognized. When the settlement start operation is recognized by the function of the operation recognition unit 222, the state ST is recorded as "24" in the timing buffer 822. When the processor 81 confirms that the state ST is recorded as "24" in the timing buffer 822, it determines as YES in ACT 101 and proceeds to ACT 102. As ACT 102, the processor 81 retrieves the timing buffer 822 that records the state ST as "24". Then, the processor 81 calculates the number of state STs of "11", that is, the number of take-out actions Na. In addition, as ACT 103, the processor 81 calculates the number of state STs of "22", that is, the number of commodity registrations Nb. Then, as ACT 104, the processor 81 confirms whether the number of take-out actions Na and the number of commodity registrations Nb are consistent.

[0178] When the customer correctly registers the commodity taken out by the taking-out action to the self-service POS terminal 11, the taking-out action number Na and the commodity registration number Nb are consistent. Therefore, when the taking-out action number Na and the commodity registration number Nb are consistent, the processor 81 determines YES in ACT 104 and ends the function as the fraud detection unit 223.

[0179] Correspondingly, when the number of removal actions Na and the number of commodity registrations Nb are inconsistent, it is considered that an unfair behavior "false registration" has been performed. Therefore, when the number of removal actions Na and the number of commodity registrations Nb are inconsistent, the processor 81 determines NO in ACT104 and proceeds to ACT105. As ACT105, the processor 81 sets the error code EC to "92". In addition, as ACT106, the processor 81 obtains the current time TM measured by the clock 84. Then, as ACT107, the processor 81 associates the time TM and the error code EC in the timing buffer 822 in which the state ST is recorded as "24" and records them. At this point, the function as the fraud detection unit 223 is terminated.

[0180] In this way, Fig.13 In the illustrated embodiment, when the number of removal actions Na and the number of commodity registrations Nb do not match, the error code EC is recorded as "92" in the sequence buffer 822. Therefore, the notification unit 224 functions to notify the illegal behavior "false registration".

[0181] exist Fig.14In ACT111, the processor 81 waits for the recognition settlement start operation to be received. When the processor 81 confirms that the state ST is "24" recorded in the timing buffer 822, it determines YES in ACT111 and proceeds to ACT112. As ACT112, the processor 81 retrieves the timing buffer 822 that records the state ST as "24". Then, the processor 81 calculates the number of state STs of "12", that is, the number of registration actions Nc. In addition, as ACT113, the processor 81 calculates the number of state STs of "22", that is, the number of commodity registrations Nd. Then, as ACT114, the processor 81 confirms whether the number of registration actions Nc and the number of commodity registrations Nd are consistent.

[0182] When the customer performs a correct registration action, the registration action count Nc and the commodity registration count Nd are consistent. Therefore, when the registration action count Nc and the commodity registration count Nd are consistent, the processor 81 determines YES in ACT 114 and terminates the function as the fraud detection unit 223.

[0183] Correspondingly, when the customer performs the registration action without aligning the barcode with the reading window, for example, the registration action number Nc is less than the product registration number Nd. Therefore, when the registration action number Nc is less than the product registration number Nd, the processor 81 determines NO in ACT 114 and proceeds to ACT 115. As ACT 115 to ACT 117, the processor 81 executes the same Fig.13 The same processing as ACT 105 to ACT 107 is performed. The function of the fraud detection unit 223 is now terminated.

[0184] In this way, Fig.14 In the illustrated embodiment, when the registration action count Nc is less than the commodity registration count Nd, the error code EC is recorded as “92” in the sequence buffer 822. Therefore, the notification of the unfair behavior “false registration” is performed by the function of the notification unit 224.

[0185] In the above embodiment, one camera 21 is configured for one self-service POS terminal 11. It is not necessary to configure a camera 21 for each self-service POS terminal 11. For example, if one camera 21 can be used to capture the customer operating two adjacent self-service POS terminals 11, the number of cameras 21 can be reduced. Figure 7 In ACT 2, the cash register number of the self-service POS terminal 11 closest to the position of the person reflected in the camera data is obtained.

[0186] In addition, for example, a plurality of cameras 21 may be configured for one self-service POS terminal 11. In this way, the blind spots of the self-service POS terminal 11 can be reduced, and improper behavior of customers can be detected with better accuracy.

[0187] In the above embodiment, the case where the notification unit 224 notifies the waiter through the waiter terminal 14 is exemplified. The notification destination is not limited to the waiter terminal 14. For example, the self-service POS terminal 11 that has committed an improper act may be used as the notification destination. In this case, for example, the light emitting unit 65 emits light of a specified color to notify the clerk that the improper act has occurred. Alternatively, a warning message is displayed on the touch panel 41 and the customer is notified that the improper act has been detected. In addition, a warning of improper behavior may be displayed or issued using sound through wireless communication with a communication terminal carried by the clerk.

[0188] In the above embodiment, the attendant terminal 14 may also include a function as the display control device 13. In this case, the operation recognition unit 222 obtains the data of the surveillance image SC from the attendant terminal 14 and recognizes the customer's operation on the self-service POS terminal 11. Alternatively, the operation recognition unit 222 may obtain the data signal output from each self-service POS terminal 11 from the communication network 15 via a router and recognize the customer's operation on the self-service POS terminal 11 based on the data signal.

[0189] The improper behavior recognition device 22 may also include a storage unit for storing the video data obtained from each camera 21. In this way, when the improper behavior of the customer is detected, the video data obtained in the storage unit can be reproduced to verify the improper behavior. When the action recognition unit 221 detects that a person is standing in front of the self-service POS terminal 11 from the video data of the camera 21, the video data is stored in the auxiliary storage device 83.

[0190] In addition, the action recognition unit 221 of the embodiment performs action recognition on the guest standing in front of the self-service POS terminal 11. However, it is assumed that there are multiple guests coming to shop, for example, friends, parents with children, etc. In this case, it is assumed that there are multiple guests standing in front of the self-service POS terminal 11. If the action recognition of all people is performed, the possibility of recognition errors increases. There is a possibility of causing trouble to the guests. If the guest standing in front of the self-service POS terminal 11 is one person, the action recognition unit 221 performs action recognition on the guest standing in front. When it is recognized that there are multiple guests (the situation where multiple guests are recognized in the camera data), it waits before the self-service POS terminal 11 is operated. When one of the guests performs an action to operate the self-service POS terminal 11 (for example, an action to start the use operation), the guest can be regarded as a shopper. In this way, one person can be designated as a shopper from among multiple guests for action recognition. A shopper designation unit may also be provided which designates a single guest as a shopper when the camera's video data identifies that the guest is operating the self-service POS terminal 11 (settlement terminal); and designates the person operating the self-service POS terminal 11 as a shopper when multiple guests are identified.

[0191] In addition, although several embodiments of the present invention are described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, changes, etc. can be made without departing from the scope of the invention. These embodiments and their variations are all included in the scope of the invention, and are included in the invention described in the scope of the claims and their equivalents.

Claims

1. An improper behavior identification device, characterized in that: include: An action recognition unit, which recognizes the actions of the shopper relative to the checkout terminal; An operation recognition unit for recognizing an operation performed on the settlement terminal by the shopper; a fraud detection unit that detects the fraudulent behavior of the shopper based on the correspondence between first status information indicating the shopper's action and a time recorded together with the first status information and second status information indicating an operation performed on the settlement terminal by the shopper and a time recorded together with the second status information; as well as Notification department, notifying the said shopper of improper behavior, The fraud detection unit detects the shopper's fraudulent behavior when the order of the operations of the second state information sequentially representing the operations recognized by the operation recognition unit conflicts with the order of the actions of the first state information sequentially representing the actions of the shopper recognized by the action recognition unit.

2. The unfair behavior identification device according to claim 1, wherein: The fraud detection unit detects fraudulent behavior of the shopper when the number of operations recognized by the operation recognition unit and the behavior of the shopper recognized by the behavior recognition unit contradict each other.

3. The unfair behavior recognition device according to claim 1 or 2, wherein: The action recognition unit recognizes the action of the shopper based on the position of a component of the payment terminal and the position of a hand of the shopper facing the payment terminal.

4. The unfair behavior recognition device according to claim 1 or 2, wherein: The operation recognition section recognizes an operation performed on the settlement terminal by the shopper based on an image converted according to the operation on the settlement terminal.

5. The unfair behavior recognition device according to claim 3, wherein: The operation recognition section recognizes an operation performed on the settlement terminal by the shopper based on an image converted according to an operation on the settlement terminal.

6. The unfair behavior identification device according to claim 1, wherein: The action recognition unit further acquires image data from a camera and stores the image data.

7. A storage medium storing a control program, wherein the control program is used to enable a computer of an unfair behavior recognition device to implement the following functions: Functionality to identify shopper actions relative to the checkout terminal; A function of identifying an operation performed on the settlement terminal by the shopper; A function of detecting improper behavior of the shopper based on a correspondence between first status information indicating an action of the shopper and a time recorded together with the first status information and second status information indicating an operation performed on the settlement terminal by the shopper and a time recorded together with the second status information; and Function to inform said shopper of improper behavior, When the order of the operations in the second state information indicating the operations in time series is inconsistent with the order of the actions in the first state information indicating the actions of the shopper in time series, the unfair behavior of the shopper is detected.

8. A method for identifying improper behavior, wherein an improper behavior identification device performs the following steps: Identify shopper actions relative to the checkout terminal; identifying an operation performed on the checkout terminal by a shopper; and Based on the correspondence between first status information indicating the actions of the shopper and the time recorded together with the first status information and second status information indicating the operations performed on the settlement terminal by the shopper and the time recorded together with the second status information, the improper behavior of the shopper is detected, When the order of the operations in the second state information indicating the operations in time series is inconsistent with the order of the actions in the first state information indicating the actions of the shopper in time series, the unfair behavior of the shopper is detected.

Images