FRAUDULENT ACTIVITY ESTIMATION DEVICE, ITS CONTROL PROGRAM, AND FRAUDULENT ACTIVITY ESTIMATION METHOD

Abstract

To provide an unauthorized action estimation device, a control program therefor, and an unauthorized action estimation method for enabling appropriate handling in accordance with the level of reliability that a customer has performed an unauthorized action.SOLUTION: In a self POS system 100, an unauthorized action estimation device 22 of an unauthorized action estimation system 200 includes: a first acquisition unit 222 that acquires operation information from a commodity registration operation on a settlement terminal by a purchaser; a behavior recognition unit 221 that recognizes a behavior of the purchaser who performs the commodity registration operation; a second acquisition unit 224 that acquires reliability of the behavior of the purchaser; and an output unit 226 that outputs result information based on the operation information and the reliability.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a misconduct estimation device, a control program therefor, and a misconduct estimation method. [Background technology]

[0002] In recent years, self-service POS (Point of Sales) terminals have been gaining attention in supermarkets and other retail stores from the perspective of reducing labor costs and taking measures to prevent the spread of infectious diseases. Self-service POS terminals are fully self-service payment terminals that allow customers to perform all operations themselves, from registering purchased items to making payments. In many cases, surveillance cameras are installed to monitor customers operating such self-service POS terminals.

[0003] In a store system equipped with such self-service POS terminals and surveillance cameras, a customer's behavior is recognized, for example, from the movement of their hand holding a purchased item, based on the captured image data captured by the surveillance camera. The customer's behavior is then used to determine whether or not they are engaging in fraudulent activity. If fraud is determined, the system may notify a store clerk or issue a warning on the self-service POS terminal. However, if a customer's hand enters the surveillance camera's blind spot or if the hand is obscured by a shadow due to lighting conditions, the customer's behavior may not be recognized correctly. This unreliable recognition result could result in a customer's normal behavior being mistakenly determined to be fraudulent. Furthermore, a customer's fraudulent behavior could be mistakenly determined to be normal. For this reason, there is a demand for ways to avoid situations such as customer disputes and a decline in trust in the store. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-41255 A Summary of the Invention [Problem to be solved by the invention]

[0005] The problem that the embodiments of the present invention aim to solve is to provide a fraudulent activity estimation device, its control program, and a fraudulent activity estimation method that can take appropriate action depending on the degree of reliability of a customer's fraudulent activity. [Means for solving the problem]

[0006] In one embodiment, the misconduct estimation device includes a first acquisition means, a behavior recognition means, a second acquisition means, and fraud estimation means; The first acquisition means acquires operation information from a product registration operation performed by a purchaser on a payment terminal. The behavior recognition means recognizes the behavior of the purchaser performing the product registration operation. The second acquisition means recognizes the behavior of the purchaser. Recognition result of High reliability Get the confidence level. The fraud inference means infers fraudulent behavior by the purchaser based on the operation information and the reliability. The output means is If fraudulent activity by a purchaser is suspected, Based on operation information and reliability This will prevent fraudulent behavior by buyers. Output the result information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal is installed. [Figure 2] FIG. 2 is a diagram for explaining the positional relationship between the self-service POS terminal and the camera. [Figure 3] FIG. 3 is a schematic diagram showing an example of a monitoring image displayed on the display of the attendant terminal. [Figure 4] FIG. 4 is a block diagram showing the main circuit configuration of the fraudulent activity inference device. [Figure 5] FIG. 5 is a schematic diagram showing an example of the data structure of the behavior file. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of a time-series buffer. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of the output table. [Figure 8] FIG. 8 is a flowchart for explaining the functions of the behavior recognition unit and the second acquisition unit. [Figure 9]FIG. 9 is a flowchart illustrating the functions of the operation recognition unit and the first acquisition unit. [Figure 10] FIG. 10 is a flowchart for explaining the function of the fraud estimation unit. [Figure 11] FIG. 11 is a flowchart for explaining the function of the output unit. [Figure 12] FIG. 12 is a schematic diagram for explaining another embodiment related to behavior recognition. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment will be described below with reference to the drawings. FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal 11 has been introduced. The system includes a self-service POS system 100 and a fraudulent activity estimation system 200. The self-service POS system 100 includes multiple self-service POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The multiple self-service POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

[0009] The self-service POS terminal 11 is a fully self-service payment terminal that allows customers to perform operations from registering purchased items to making payments themselves. A customer may also be referred to as a purchaser, consumer, or client. The POS server 12 is a server computer that centrally controls the operation of each self-service POS terminal 11. The display control device 13 is a controller that generates a monitoring image SC (see FIG. 3) for each self-service POS terminal 11 based on data signals output from each self-service POS terminal 11 and displays the image on the display device of the attendant terminal 14. The attendant terminal 14 is a terminal used by store clerks known as attendants to monitor the status of each self-service POS terminal 11. The attendant terminal 14 is equipped with a display such as a liquid crystal display or an organic EL display. The attendant terminal 14 divides the display screen into multiple sections, and displays a different monitoring image SC for the attendant terminal 14 for each section. The attendant terminal 14 is an example of a store clerk terminal. Conventionally known terminals can be used as is for this self-service POS system 100.

[0010] The fraudulent activity estimation system 200 includes a plurality of cameras 21 and a fraudulent activity estimation device 22. The plurality of cameras 21 correspond one-to-one to the plurality of self-service POS terminals 11. The cameras 21 are used to capture images of customers operating the corresponding self-service POS terminals 11.

[0011] The fraudulent activity estimation device 22 has functions as a behavior recognition unit 221, an operation recognition unit 222, a first acquisition unit 223, a second acquisition unit 224, a fraud estimation unit 225, and an output unit 226. The behavior recognition unit 221 has a function of recognizing the behavior of a customer performing a product registration operation in the self-service POS terminal 11 based on the photographic data output from each camera 21. The behavior recognition unit 221 can also be referred to as a behavior recognition means. The operation recognition unit 222 has a function of recognizing a product registration operation by a customer in the self-service POS terminal 11 based on the data of the surveillance image SC output from the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be referred to as an operation recognition means. The first acquisition unit 223 has a function of acquiring operation information from the recognized product registration operation. The first acquisition unit 223 can also be referred to as a first acquisition means. The second acquisition unit 224 has a function of acquiring the reliability of the recognition result based on the recognition result by the behavior recognition unit 221. The second acquisition unit 224 can also be referred to as second acquisition means. The fraud estimation unit 225 has a function of inferring fraudulent behavior by a customer based on the operation information from the first acquisition unit 223 and the reliability from the second acquisition unit 224. The fraud estimation unit 225 can also be referred to as fraud estimation means. The output unit 226 has a function of outputting inferred result information. The output unit 226 can also be referred to as output means.

[0012] 2 is a diagram for explaining the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external configuration of the self-service POS terminal 11 will be described.

[0013] The self-service POS terminal 11 comprises a main body 40 placed on the floor and a bagging stand 50 placed beside the main body 40. A touch panel 41 is attached to the top of the main body 40. The touch panel 41 is composed of a display and a touch sensor. The touch panel 41 is an example of a display unit. The display is a device for displaying various screens to an operator who operates the self-service POS terminal 11. The touch sensor is a device for detecting touch inputs made by the operator on the screen. In the self-service POS terminal 11, the operator is usually a customer.

[0014] The main body 40 has a basket stand 60 in the center of the side opposite the side where the bagging stand 50 is installed. The basket stand 60 is for customers coming from the sales floor to place baskets containing purchased items. Customers work by standing on the front side of the main body 40 in FIG. 2 so that they can see the screen of the touch panel 41. Therefore, from the customer's perspective, the basket stand 60 is on the right side of the main body 40, and the bagging stand 50 is on the left side. In this embodiment, the side where the customer stands is the front of the main body 40, the side where the bagging stand 50 is installed is the left side of the main body 40, and the side where the basket stand 60 is installed is the right side of the main body 40.

[0015] The main body 40 is equipped with a scanner, card reader, receipt printer, cash processing machine, etc. The front of the main body 40 is formed with a scanner reading window 42, a card insertion slot 43, a receipt issuing slot 44, a coin insertion slot 45, a coin dispensing slot 46, a bill insertion slot 47, and a bill dispensing slot 48. A communication cable extends from the right side of the main body 40 to the outside, and a handy scanner 61 is connected to the tip of this communication cable. Although not shown, the main body 40 is also equipped with a reader / writer for electronic money media.

[0016] A display pole 64 is attached to the upper surface of the main body 40. The display pole 64 has a light-emitting unit 65 at its tip. The light-emitting unit 65 selectively emits light, for example, blue or red. The display pole 64 displays the status of the self-service POS terminal 11, such as waiting, operating, calling, error, or fraudulent activity, depending on the color of light emitted by the light-emitting unit 65. The display pole 64 may also display the status of the self-service POS terminal 11 by flashing the light-emitting unit 65.

[0017] The bagging table 50 has a structure in which a bag holder 52 is attached to its upper part. The bag holder 52 has a pair of holding arms 53, which hold plastic bags provided by the store or shopping bags brought by customers, so-called "my bags," etc.

[0018] Next, the positional relationship between the self-service POS terminal 11 and the camera 21 will be described. As shown in FIG. 2, the camera 21 is installed in a position where it can photograph a customer standing in front of the self-service POS terminal 11 and facing the main body 40, bagging table 50, basket table 60, and other components from above.

[0019] A customer standing in front of the self-service POS terminal 11 first places a basket containing purchased items on the basket stand 60 on the right side, and then places a shopping bag or a personal bag on the holding arm 53 on the left side. Next, the customer operates the touch panel 41 according to the guidance displayed on the touch panel 41 to declare that they want to start using the self-service POS terminal 11.

[0020] The customer then picks up each item to purchase from the basket placed on the basket stand 60. If the purchased item has a barcode, the customer registers the item by holding the barcode over the reading window 42 and having it read by the scanner. If the purchased item does not have a barcode, the customer registers the item by operating the touch panel 41 to select the item from a list of items without barcodes. After registering the items, the customer places them in a shopping bag, their own bag, or the like.

[0021] After registering all of the purchased items, the customer operates touch panel 41 to select a payment method. For example, if cash payment is selected, the customer inserts bills or coins into bill slot 47 or coin slot 45, and removes the change from bill dispensing outlet 48 or coin dispensing outlet 46. For example, if electronic money payment is selected, the customer holds the electronic money medium over the reader / writer. For example, if credit card payment is selected, the customer inserts the credit card into card slot 43. After completing the payment in this way, the customer receives the receipt issued from receipt issuing port 44 and leaves the store with the shopping bag or personal bag removed from holding arm 53.

[0022] That is, the camera 21 is installed in a position in front of the self-service POS terminal 11 so as to be able to capture the hand movements of the customer who is behaving as described above.

[0023] Fig. 3 is a schematic diagram showing an example of a monitoring image SC displayed on the display of the attendant terminal 14. As described above, the monitor images SC for the multiple self-service POS terminals 11 are displayed separately on the display of the attendant terminal 14. Fig. 3 shows an example of a monitoring image SC for one of the self-service POS terminals 11. The configuration of the monitoring image SC for the other self-service POS terminals 11 is similar, so a description thereof will be omitted here.

[0024] As shown in FIG. 3, the monitoring image SC includes a register number column 71, a terminal status column 72, an error information column 73, a declaration information column 74, a details column 75, and a total column .

[0025] The cash register number column 71 is a column for displaying the cash register number. The cash register number is a series of numbers assigned to each self-service POS terminal 11 so as to identify each self-service POS terminal 11 individually. The cash register number is identification information for identifying each self-service POS terminal 11.

[0026] The terminal status field 72 is a field for displaying the operating status of the self-service POS terminal 11. For example, the terminal status field 72 displays one of the following operating statuses: "Waiting," "Start of use," "Registering," "Payment started," and "Payment in progress."

[0027] "Waiting" is the state from when the previous customer has finished paying until the next customer is declared ready to use the store. An initial image is displayed on the touch panel 41 of the self-service POS terminal 11 in the "Waiting" state. The initial image is an image that includes touch buttons that allow the customer to select whether to use the store's provided shopping bags or their own bag, for example.

[0028] "Start of use" is the state in which a customer standing in front of the self-service POS terminal 11 declares the start of use for payment. The customer selects on the initial image whether to use a plastic bag or a personal bag. This selection operation declares the start of use. In response to this selection operation, the operating state of the self-service POS terminal 11 becomes "start of use."

[0029] "Registering" is a state in which the self-service POS terminal 11 is accepting the customer's operation to register the purchased item. When the first purchased item is registered, the self-service POS terminal 11 enters the "registering" state. After that, the self-service POS terminal 11 maintains the "registering" state until the transition to payment is declared.

[0030] "Payment Start" is the state in which a customer who has finished registering the items to be purchased declares that they want to move on to payment. The touch panel 41 of the self-service POS terminal 11 in the "Registering" state displays the "Checkout" soft key. The customer who has finished registering the items to be purchased touches the "Checkout" soft key. This operation declares that they want to move on to payment. In response to this operation, the operating state of the self-service POS terminal 11 changes to "Payment Start."

[0031] "Payment in progress" is a state in which payment processing such as cash payment, electronic money payment, credit card payment, etc. For example, when a bill or coin is inserted into the bill insertion slot 47 or the coin insertion slot 45, the operating state of the self-service POS terminal 11 becomes "payment in progress." Then, when the payment processing is completed, the operating state of the self-service POS terminal 11 returns to "standby."

[0032] The error information field 73 is a field for displaying error information that has occurred in the self-service POS terminal 11. Error information includes communication errors, out-of-receipt errors, etc. The declaration information field 74 is a field for displaying the contents of the customer's declaration. For example, if a customer selects their own bag, "No bag needed" is displayed, indicating that a plastic bag is not required.

[0033] The details column 75 is a column for displaying detailed information of purchased items registered at the self-service POS terminal 11. The detailed information includes, for example, the product name, number of items, and price of the purchased items. The total column 76 is a column for displaying total information of purchased items registered at the self-service POS terminal 11. The total information includes the total number of items, the total price, the amount inserted, change, etc.

[0034] It should be noted that the configuration of the monitoring image SC is not limited to this. Columns for displaying other items may be arranged. Furthermore, the items of text data displayed in FIG. 3 are not limited to this. Text data of other items may be displayed.

[0035] 4 is a block diagram showing the main circuit configuration of the misconduct estimation device 22. The misconduct estimation device 22 includes a processor 81, a main memory 82, an auxiliary storage device 83, a clock 84, a speaker 85, a camera interface 86, a communication interface 87, and a system bus 88. The system bus 88 includes an address bus, a data bus, etc. The misconduct estimation device 22 configures a computer by connecting the processor 81, the main memory 82, the auxiliary storage device 83, the clock 84, the speaker 85, the camera interface 86, and the communication interface 87 via the system bus 88.

[0036] The processor 81 corresponds to the central part of the computer. The processor 81 controls each part to realize various functions of the misconduct estimation device 22 in accordance with an operating system or an application program. The processor 81 is, for example, a CPU (Central Processing Unit).

[0037] The main memory 82 corresponds to the main storage portion of the computer. The main memory 82 includes a nonvolatile memory area and a volatile memory area. The main memory 82 stores an operating system or application programs in the nonvolatile memory area. The main memory 82 stores data required for the processor 81 to execute processes for controlling each part in the volatile memory area. This type of data may also be stored in the nonvolatile memory area. The main memory 82 uses the volatile memory area as a work area where data is rewritten by the processor 81 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0038] The auxiliary storage device 83 corresponds to the auxiliary storage portion of the computer. As the auxiliary storage device 83, for example, a well-known storage device such as an SSD (Solid State Drive), an HDD (Hard Disc Drive), or an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory) is used alone or in combination. The auxiliary storage device 83 stores data used by the processor 81 when performing various processes, data generated by the processes in the processor 81, etc. The auxiliary storage device 83 may also store application programs.

[0039] The application programs stored in the main memory 82 or the auxiliary storage device 83 include a control program, which will be described later. There are no particular restrictions on the method for installing the control program into the main memory 82 or the auxiliary storage device 83. The control program can be recorded on a removable recording medium, or can be distributed by communication via a network and installed in the main memory 82 or the auxiliary storage device 83. The recording medium can be in any form, such as a CD-ROM or memory card, as long as it can store the program and is readable by the device.

[0040] The clock 84 functions as a time information source for the fraudulent activity estimation device 22. The processor 81 obtains the current date and time based on the time information kept by the clock 84.

[0041] The speaker 85 is an output device for outputting sound data, which includes sounds, voices, and the like.

[0042] The camera interface 86 is an interface for communicating with each camera 21. The photographed data output from each camera 21 is input into the fraudulent activity estimation device 22 via the camera interface 86. The photographed data includes photographed video, photographed images, etc. of a customer operating the self-service POS terminal 11 corresponding to the camera 21.

[0043] The communication interface 87 is an interface for performing data communication in accordance with a communication protocol between the self-service POS terminal 11, the POS server 12, the display control device 13, etc. For example, image data output from the display control device 13 is imported into the fraudulent activity estimation device 22 via the communication interface 87. The image data is data of the surveillance image SC generated for each self-service POS terminal 11.

[0044] The misconduct inference device 22 configured as above allocates part of the volatile memory area in the main memory 82 as an area for a behavior file 821 (see FIG. 5), a time-series buffer 822 (see FIG. 6), and an output table 823 (see FIG. 7). The misconduct inference device 22 then creates the behavior file 821, the time-series buffer 822, and the output table 823 in this area.

[0045] 5 is a schematic diagram showing an example of the data structure of the behavior file 821. As shown in FIG. 5, the behavior file 821 is a data file that records, in association with each other, a time TM, a behavior status AST, a recognition rate RP, and a frame image for each register number that identifies the self-checkout POS terminal 11. The time TM is the time when the behavior status AST is acquired. The behavior status AST represents the state in which the behavior of the customer is recognized by the function of the behavior recognition unit 221 of the processor 81, which will be described later. In this embodiment, the behavior of the customer is a picking behavior and a bagging behavior.

[0046] The take-out action is an action of taking out a purchased item from a basket placed on the basket stand 60. For example, when the skeleton of one or both hands moves to the right side of the main body 40 and a movement of lifting up the purchased item is detected, the processor 81 recognizes that a take-out action has occurred.

[0047] The bagging behavior is the behavior of placing registered purchased items into a shopping bag or a personal bag on the bagging table 50. For example, when the skeleton of the hand that performed the registered behavior moves to the left side of the main body and a movement of placing the purchased items into a shopping bag or a personal bag is detected, the processor 81 recognizes that a bagging behavior has occurred.

[0048] The recognition rate RP is a numerical value calculated based on the recognition rate at which the processor 81 recognizes the customer's behavior included in the captured image as a pick-up behavior and the recognition rate at which the processor 81 recognizes the customer's behavior as a bagging behavior. The recognition rate RP is, for example, a percentage. The frame images are captured images taken by the camera 21. The frame images are recorded in the behavior file 821 in the order in which they were acquired. Note that the items displayed in Figure 5 are not limited to these. Other items may also be displayed. The content of the text data displayed in Figure 5 is an example.

[0049] Fig. 6 is a schematic diagram showing an example of the time-series buffer 822. As shown in Fig. 6, the time-series buffer 822 has an area for describing the start time STM, the end time FTM, the status ST or the output code OC (see Fig. 7), and the reliability CD in association with each other for each register number that identifies the self-checkout POS terminal 11. The start time STM is the time TM at which the behavior status AST is earliest written in the behavior file 821. The start time STM is the time at which the operation status HST is acquired. The start time STM is the time at which the output code OC is acquired. The end time FTM is the time TM at which the behavior status AST is latest written in the behavior file 821. The status ST includes an action status AST and an operation status HST. The operation status HST indicates a state in which a product registration operation by a customer to the self-checkout POS terminal 11 is recognized by the function of an operation recognition unit 222 of the processor 81, which will be described later. The reliability CD is the reliability of the recognition result that the processor 81 recognizes as the customer's behavior. The reliability CD is a numerical value calculated based on the recognition rate RP. The reliability CD is, for example, a percentage. The reliability CD is an example of the reliability of the purchaser's behavior.

[0050] The time-series buffer 822 describes the status ST or output code OC and reliability CD in order of earliest start time STM. In this embodiment, when an action status AST is described as the status ST, the start time STM, end time FTM and reliability CD are described. When an operation status HST or output code OC is described as the status ST, the start time STM is described, and for example, NULL values ​​are described for the end time FTM and reliability CD. Note that the items displayed in FIG. 6 are not limited to these. Other items may also be displayed.

[0051] Fig. 7 is a schematic diagram showing an example of the output table 823. As shown in Fig. 7, the output table 823 is a data table in which the output code OC, the threshold value of the reliability CD of the action status AST "11", the threshold value of the reliability CD of the action status AST "12", and the output data are described in association with each other. The output code OC is unique output identification information identified by the reliability CD threshold of the action status AST "11", the reliability CD threshold of the action status AST "12", and the output data in the same row. The action status AST "11" indicates a pick-up action. The action status AST "12" indicates a bag-filling action. The threshold for the reliability CD of the behavior status AST "11" and the threshold for the reliability CD of the behavior status AST "12" can be set arbitrarily. The threshold may be a fixed numerical value in the fraudulent activity estimation device 22, or may be changeable to a desired threshold value by the store. In this embodiment, the store sets the threshold for the reliability CD of the behavior status AST "11" and the threshold for the reliability CD of the behavior status AST "12" in advance. The threshold can also be said to be a numerical value that allows the store to output output data of the same row, a reference value for outputting output data, etc. The threshold is an example of a predetermined condition. The output data is data that is output to the self-POS terminal 11 or the attendant terminal 14 when the reliability CD threshold of the action status AST "11" and the reliability CD threshold of the action status AST "12" in the same row are satisfied. The output data is, for example, text data for the self-POS terminal, text data for the attendant terminal, sound data, color data, etc.

[0052] The text data for the self-service POS terminal may be, for example, "There is a purchased item that was not registered." The text data for the self-service POS terminal may contain any content that notifies the customer of fraudulent activity. The text data for the self-service POS terminal is an example of result information. The text data for the self-service POS terminal is an example of information that notifies the customer of fraudulent activity. An example of the text data for the attendant terminal is "Fraudulent activity has occurred at register No. X." The text data for the attendant terminal may contain content that notifies the attendant of a customer's fraudulent activity. The text data for the attendant terminal is an example of result information. The text data for the attendant terminal is an example of information that notifies the attendant of fraudulent activity. The sound data may be, for example, a continuous sound or an intermittently repeated sound. The sound data may be, for example, a voice saying, "There is a purchase item that was not registered." The sound data may be any sound that notifies a customer or an attendant of the customer's fraudulent activity. The sound data is an example of result information. The sound data is an example of a sound that notifies of fraudulent activity. The color data is, for example, data representing the emitted color. The color data may be any color that notifies the attendant of a customer's fraudulent behavior. The color data is an example of result information. In this embodiment, the store sets the output data in advance. The output data may include, for example, data regarding the output destination and data regarding the output method.

[0053] As shown in FIG. 7, for example, if the reliability CD of the behavioral status AST "11" is 95 or higher and the reliability CD of the behavioral status AST "12" is 85 or higher, text data and sound data for the self-service POS terminal corresponding to output codes OC "91" and "93" are output as output data. Depending on the setting of the threshold for the reliability CD of the behavioral status AST "11" and the threshold for the reliability CD of the behavioral status AST "12," one piece of output data may be output, or a combination of multiple pieces of output data may be output. Note that the items displayed in FIG. 7 are not limited to these. Other items may also be displayed. The content of the text data displayed in FIG. 7 is an example.

[0054] The functions of the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud estimation unit 225, and output unit 226 of the fraudulent activity inference device 22 are functions provided for each self-service POS terminal 11. The functions of the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud estimation unit 225, and output unit 226 for other self-service POS terminals 11 are similar. The functions of the behavior recognition unit 221, operation recognition unit 222, first acquisition unit 223, second acquisition unit 224, fraud estimation unit 225, and output unit 226 of the fraudulent activity inference device 22 are realized by a processor 81 and a control program that controls the processor 81.

[0055] 8 to 11 are flowcharts showing the control procedure of the main parts of the processor 81 in the fraudulent activity estimation device 22. Below, these flowcharts will be used to explain the main operations of the system of a store in which the self-service POS terminal 11 has been introduced. Note that the operations explained below are just an example. The procedures are not particularly limited as long as similar results can be obtained.

[0056] FIG. 8 is a flowchart for explaining the functions of the behavior recognition unit 221 and the second acquisition unit 224. The processor 81 waits to recognize a customer as ACT1. The camera 21 is installed in a position where it can capture an image from above of a customer standing in front of the self-service POS terminal 11. When the processor 81 detects from the image captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, it determines that the customer has been recognized.

[0057] When the processor 81 recognizes the customer, it determines YES in ACT1 and proceeds to ACT2. In ACT2, the processor 81 acquires the register number of the self-service POS terminal 11. Each camera 21 corresponds one-to-one with each self-service POS terminal 11. Therefore, the processor 81 identifies the self-service POS terminal 11 from the identification information of the camera 21 capturing an image of the customer standing in front of the self-service POS terminal 11, and acquires the register number of that self-service POS terminal 11.

[0058] The processor 81 starts acquiring frame images captured by the camera 21 as ACT3. The processor 81 stores the frame image in the behavior file 821 every time it acquires a frame image.

[0059] The processor 81 estimates the skeleton of a person, i.e., a customer, from frame images captured by the camera 21 as ACT4. Skeleton estimation can be achieved even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the customer's take-out or bagging behavior from the hand movements obtained by the skeleton estimation and the temporal changes in the positional relationship between the hand and a region of interest (ROI), such as the main body 40 of the self-service POS terminal 11, the bagging table 50, or the basket table 60, and calculates a recognition rate RP for each. Specifically, the processor 81 calculates the recognition rate for the take-out behavior and the recognition rate for the bagging behavior from the hand movements. The processor 81 determines the highest recognition rate among these recognition rates as the recognition rate RP. For example, if the recognition rate for the take-out behavior is 98 percent and the recognition rate for the bagging behavior is 2 percent, the recognition rate RP is 98 percent. The processor 81 recognizes the take-out behavior for which a recognition rate RP of 98 percent is calculated as the customer's behavior. The process of calculating the recognition rate RP based on hand movements is an existing process that is well known, so a detailed description thereof will be omitted.

[0060] In ACT 5, the processor 81 checks whether or not a take-out action has been recognized based on the processing in ACT 4. If a take-out action has been recognized by the function of the action recognition unit 221, the processor 81 determines YES in ACT 5 and proceeds to ACT 6.

[0061] In ACT6, the processor 81 sets the action status AST to "11." After that, the processor 81 proceeds to the processing of ACT12.

[0062] In ACT 12, processor 81 acquires the current time TM measured by clock 84. In ACT 13, processor 81 associates and writes the time TM, the action status AST of "11," and the recognition rate RP calculated in ACT 4 in the action file 821 in which the register number acquired in ACT 2 is set, in association with the frame image saved in ACT 3. Then, processor 81 returns to ACT 3.

[0063] If the take-out action is not recognized, the processor 81 determines NO in ACT5 and proceeds to ACT7.

[0064] In ACT 7, the processor 81 checks whether or not a bagging action has been recognized based on the processing in ACT 4. If a bagging action has been recognized by the function of the action recognition unit 221, the processor 81 determines YES in ACT 7 and proceeds to ACT 8.

[0065] In ACT8, the processor 81 checks whether "11" is written as the action status AST along with the start time STM, end time FTM, and reliability CD in the time series buffer 822 in which the register number obtained in the processing of ACT2 is set.

[0066] If "11" is not written as the behavior status AST in the time-series buffer 822, the processor 81 determines NO in ACT8 and proceeds to ACT9.

[0067] The processor 81 refers to the action file 821 as ACT9 using the function of the second acquisition unit 224, and calculates the reliability CD based on the recognition rate RP for which "11" is written as the action status AST. For example, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP for which "11" is written as the action status AST. The processor 81 sets this highest recognition rate RP as the reliability CD of the take-out action. A high reliability CD of the take-out action means that the reliability of the recognition result recognized by the processor 81 as the take-out action is high, i.e., the reliability that the customer performed the take-out action is high. A low reliability CD of the take-out action means that the reliability of the recognition result recognized by the processor 81 as the take-out action is low, i.e., the reliability that the customer performed the take-out action is low.

[0068] The processor 81 writes "11" as the behavior status AST, along with the start time STM, end time FTM, and reliability CD, in the time series buffer 822 as ACT10. The start time STM is the time TM at which "11" is written earliest as the behavior status AST in the behavior file 821. The end time FTM is the time TM at which "11" is written latest as the behavior status AST in the behavior file 821. The processor 81 then proceeds to ACT11.

[0069] If "11" is written as the action status AST in the time-series buffer 822, the processor 81 determines YES in ACT8, skips the processing of ACT9 and ACT10, and proceeds to ACT11.

[0070] The processor 81 sets the action status AST to "12" as ACT11.

[0071] In ACT 12, processor 81 acquires the current time TM measured by clock 84. In ACT 13, processor 81 associates and writes the time TM, the action status AST of "12," and the recognition rate RP calculated in the processing of ACT 4 in the action file 821, corresponding to the frame image saved in the processing of ACT 3. Then, processor 81 returns to ACT 3.

[0072] If the processor 81 does not recognize the bagging behavior, it determines NO in ACT7 and proceeds to ACT14.

[0073] The processor 81 checks whether or not "12" is written as the action status AST in the time-series buffer 822 as ACT14, together with the start time STM, end time FTM, and reliability CD.

[0074] If "12" is not written as the behavior status AST in the time-series buffer 822, the processor 81 determines NO in ACT14 and proceeds to ACT15.

[0075] The processor 81 refers to the action file 821 using the function of the second acquisition unit 224 as ACT15, and calculates the reliability CD based on the recognition rate RP for which "12" is written as the action status AST. For example, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP for which "12" is written as the action status AST. The processor 81 sets this highest recognition rate RP as the reliability CD of the bagging action. A high reliability CD of the bagging action means that the reliability of the recognition result recognized by the processor 81 as the bagging action is high, i.e., the reliability that the customer performed the bagging action is high. A low reliability CD of the bagging action means that the reliability of the recognition result recognized by the processor 81 as the bagging action is low, i.e., the reliability that the customer performed the bagging action is low.

[0076] The processor 81 writes "12" as the behavior status AST together with the start time STM, end time FTM, and reliability CD in the time series buffer 822 as ACT16. The start time STM is the time TM at which "12" is written earliest as the behavior status AST in the behavior file 821. The end time FTM is the time TM at which "12" is written latest as the behavior status AST in the behavior file 821. With this, the processor 81 terminates its functions as the behavior recognition unit 221 and the second acquisition unit 224.

[0077] If "12" is written as the behavior status AST in the time-series buffer 822, the processor 81 judges YES in ACT14, skips the processing of ACT15 and ACT16, and terminates the functions of the behavior recognition unit 221 and the second acquisition unit 224.

[0078] Usually, a customer registers data of purchased items in the self-service POS terminal 11 by sequentially repeating the actions of picking up and packing items in the self-service POS terminal 11. Therefore, the action status AST is written in chronological order in the time-series buffer 822 in the order of "11" and "12."

[0079] Thereafter, when it is detected again from the image captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, the processor 81 executes the processes of ACT2 to ACT16 in the same manner as described above.

[0080] FIG. 9 is a flowchart for explaining the functions of the operation recognition unit 222 and the first acquisition unit 223. The processor 81 waits for the start of use to be declared to the self-service POS terminal 11 as ACT21. When the start of use is declared, "Start of use" is displayed in the terminal status field 72 of the monitoring image SC corresponding to the self-service POS terminal 11. The processor 81 checks whether it can recognize the words "Start of use" from the terminal status field 72 of the monitoring image SC acquired via the display control device 13. If the words "Start of use" can be recognized, the processor 81 recognizes that the start of use has been declared by the function of the operation recognition unit 222.

[0081] When processor 81 recognizes that the start of use has been declared, it determines YES in ACT21 and proceeds to ACT22. In ACT22, processor 81 acquires the register number of self-checkout POS terminal 11. The register number is displayed in register number column 71 of monitoring image SC. Processor 81 recognizes the characters of the register number from register number column 71 of monitoring image SC acquired via display control device 13, and acquires the characters as the register number.

[0082] The processor 81 sets the operation status HST to "21" in ACT23. The operation status HST "21" indicates a start of use operation. The processor 81 acquires the current time TM measured by the clock 84 in ACT24. The processor 81 associates the time TM as the start time STM with "21" as the operation status HST in the time series buffer 822 in which the register number acquired in the processing of ACT22 is set in ACT25. The processor 81 also enters NULL values ​​in the end time FTM and reliability CD.

[0083] Therefore, when a customer standing in front of the self-service POS terminal 11 makes a declaration operation to start using the terminal, "21" is first written as the operation status HST together with the time TM in the time series buffer 822 corresponding to the self-service POS terminal 11.

[0084] As ACT26, the processor 81 starts recognizing an operation on the self-checkout POS terminal 11. Specifically, the processor 81 recognizes, for example, a product registration operation, a payment start operation, etc., from the transition of information obtained by character recognition of the monitoring image SC acquired via the display control device 13.

[0085] For example, processor 81 recognizes that a product registration operation has been performed when detailed information such as the product name, quantity, and price of the purchased product is added to detail column 75. For example, processor 81 recognizes that a payment start operation has been performed when the display in terminal status column 72 switches to "Payment Start."

[0086] The processor 81 waits for recognition of a product registration operation or a payment start operation in ACT27 or ACT28.

[0087] In the standby state of ACT27 or ACT28, if the operation recognition unit 222 recognizes a product registration operation, the processor 81 determines YES in ACT27 and proceeds to ACT29.

[0088] The processor 81 sets the operation status HST to "22" using the function of the first acquisition unit 223 as ACT29. The operation status HST "22" indicates a product registration operation. The processor 81 acquires the current time TM measured by the clock 84 using the function of the first acquisition unit 223 as ACT30. The processor 81 associates the time TM as the start time STM with "22" as the operation status HST and writes it in the time series buffer 822 as ACT31. The processor 81 writes NULL values ​​for the end time FTM and the reliability CD. The operation status HST is an example of operation information. The current time TM is an example of operation information. Thereafter, the processor 81 returns to the standby state of ACT27 or ACT28.

[0089] In the standby state of ACT27 or ACT28, when the payment start operation is recognized by the function of the operation recognition unit 222, the processor 81 determines YES in ACT28 and proceeds to ACT 32. The processor 81 ends the operation recognition for the self-checkout POS terminal 11 in ACT32.

[0090] The processor 81 sets the operation status HST to "23" as ACT33. The operation status HST "23" indicates a payment start operation. The processor 81 acquires the current time TM measured by the clock 84 as ACT34. The processor 81 associates the time TM as the start time STM with "23" as the operation status HST and writes it in the time series buffer 822 as ACT35. The processor 81 also writes NULL values ​​for the end time FTM and the reliability CD. With this, the processor 81 terminates its functions as the operation recognition unit 222 and the first acquisition unit 223.

[0091] Usually, a customer performs operations on the self-service POS terminal 11 in the order of start of use, product registration, and payment start. Therefore, the operation statuses HST are written in the time-series buffer 822 in the order of "21", "22", and "23".

[0092] Thereafter, when the operation to start use of the self-checkout POS terminal 11 is detected again from the data of the monitoring image SC, the processor 81 executes the processes of ACT22 to ACT35 in the same manner as described above.

[0093] FIG. 10 is a flowchart for explaining the function of the fraud estimation unit 225. In this embodiment, the processor 81 executes the process shown in FIG. 10 for each purchased item. In ACT41, the processor 81 checks whether or not the action status AST is "12", i.e., whether a bagging action is described, in the time-series buffer 822. If "12" is described as the action status AST, the processor 81 determines YES in ACT41 and proceeds to ACT42.

[0094] The processor 81 checks whether "22" is written as the operation status HST in ACT42, i.e., whether a product registration operation is written. Specifically, the processor 81 checks whether "22" is written as the operation status HST in association with the time TM as the start time STM, one time before the start time STM at which "12" is written as the action status AST. If "22" is written as the operation status HST, the processor 81 determines YES in ACT42 and terminates the function as the fraud estimation unit 225. Note that if "22" is written as the operation status HST, the time-series buffer 822 will be written in chronological order as the action status AST "11", the operation status HST "22", and the action status AST "12".

[0095] If "22" is not written as the operation status HST, the processor 81 determines NO in ACT 42 and proceeds to ACT 43. Note that if "22" is not written as the operation status HST, the time-series buffer 822 will chronologically write the action status AST "11" and then the action status AST "12" in that order.

[0096] The processor 81 checks in the order of output codes OC whether the reliability CD of the retrieval action in the same row as the action status AST "11" written in the time-series buffer 822 as ACT43, that is, the reliability CD of the retrieval action, satisfies the threshold of the reliability CD of the action status AST "11" written in the output table 823. If the reliability CD of the retrieval action written in the time-series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "11" written in the output table 823, the processor 81 determines NO in ACT43 and proceeds to ACT45. The processing of ACT45 will be described later.

[0097] If the reliability CD of the retrieval action described in the time-series buffer 822 satisfies the threshold of the reliability CD of the action status AST "11" described in the output table 823, the processor 81 determines YES in ACT43 and proceeds to ACT44.

[0098] The processor 81 checks whether the reliability CD of the packing action in the same row as the action status AST "12" written in the time-series buffer 822 as ACT44, i.e., the reliability CD of the packing action, satisfies the threshold of the reliability CD of the action status AST "11" confirmed in the processing of ACT43 and the threshold of the reliability CD of the action status AST "12" in the same row, written in the output table 823. If the reliability CD of the packing action written in the time-series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "12" written in the output table 823, the processor 81 determines NO in ACT44 and proceeds to ACT45.

[0099] The processor 81 records the behavior file 821 and the time-series buffer 822 as a log in ACT45. For example, the processor 81 may store the behavior file 821 and the time-series buffer 822 in a part of a non-volatile memory area in the main memory 82. In this case, the main memory 82 is an example of a storage unit. Thereafter, the processor 81 proceeds to ACT49. The processing of ACT49 will be described later.

[0100] If the reliability CD of the bagging action described in the time-series buffer 822 satisfies the threshold of the reliability CD of the action status AST "12" described in the output table 823, the processor 81 determines YES in ACT44 and proceeds to ACT46.

[0101] The processor 81 acquires the output code OC as ACT46. The processor 81 acquires the current time TM measured by the clock 84 as ACT47. The processor 81 associates the time TM as the start time STM with the output code OC acquired in the processing of ACT46 and writes them in the time series buffer 822 as ACT48. The processor 81 also writes NULL values ​​for the end time FTM and the reliability CD.

[0102] In ACT 49, the processor 81 checks whether the reliability CD of the picking action and the packing action has been compared with the threshold value for all output codes OC in the output table 823. If the reliability CD of the picking action and the packing action has not been compared with the threshold value for all output codes OC, the processor 81 determines NO in ACT 49 and returns to ACT 43. Thereafter, the processor 81 executes the processes in ACT 43 to ACT 48 in the same manner as described above.

[0103] When the comparison of the reliability of the picking action and the bagging action with the threshold value has been executed for all the output codes OC, the processor 81 determines YES in ACT49 and ends the function as the fraud estimation unit 225.

[0104] In this way, when "22" is not written as the operation status HST in the time-series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time-series buffer 822 satisfy the thresholds of the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12" written in the output table 823, respectively, the processor 81 infers that the reliability of the action of bagging the product that the customer picked up but did not perform the product registration operation is high. That is, the processor 81 infers that the customer's action is fraudulent using the function of the fraud inference unit 225. In this case, the processor 81 writes in the time-series buffer 822 an output code OC that satisfies the thresholds of the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12".

[0105] If "22" is not written as the operation status HST in the time-series buffer 822, and the reliability CD of the take-out action written in the time-series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "11" written in the output table 823, and / or the reliability CD of the bagging action written in the time-series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "12" written in the output table 823, the processor 81 infers that the reliability of the bagging action of the product that the customer took out but did not perform the product registration operation is low. In this case, the processor 81 records the action file 821 and the time-series buffer 822 as a log.

[0106] FIG. 11 is a flowchart for explaining the function of the output unit 226. In ACT51, the processor 81 waits for the output code OC to be written in the time series buffer 822. When the output code OC is written in the time series buffer 822, the processor 81 determines NO in ACT51 and proceeds to ACT52. In ACT52, the processor 81 checks whether the output code OC is "91".

[0107] If the output code OC is "91," the processor 81 determines YES in ACT52 and proceeds to ACT53. In ACT53, the processor 81 uses the function of the output unit 226 to output the first output command from the communication interface 87 to the self-service POS terminal 11. Specifically, the processor 81 obtains the register number from the time-series buffer 822 in which "91" is written as the output code OC. The processor 81 refers to the output table 823 to obtain the output data on the same line as the output code OC "91," i.e., the text data for the self-service POS terminal. The first output command includes the register number and the text data for the self-service POS terminal.

[0108] The self-checkout POS terminal 11 displays the text data for the self-checkout POS terminal on the touch panel 41 of the self-checkout POS terminal 11 identified by the register number included in the first output command. After that, the processor 81 proceeds to ACT54.

[0109] If the output code OC is not "91", the processor 81 determines NO in ACT 52 and proceeds to ACT 54. That is, after processing ACT 53 or if the output code OC is not "91", the processor 81 checks in ACT 54 whether the output code OC is "92".

[0110] If the output code OC is "92," the processor 81 determines YES in ACT54 and proceeds to ACT55. In ACT55, the processor 81 uses the function of the output unit 226 to output the second output command from the communication interface 87 to the attendant terminal 14 via the display control device 13. Specifically, the processor 81 obtains the register number from the time-series buffer 822 in which "92" is written as the output code OC. The processor 81 refers to the output table 823 to obtain the output data on the same line as the output code OC "92," i.e., the text data for the attendant terminal. The second output command includes the register number and the text data for the attendant terminal.

[0111] The display control device 13 displays the attendant terminal text data on the monitoring image SC of the attendant terminal 14 identified by the register number included in the second output command. After that, the processor 81 proceeds to ACT56.

[0112] If the output code OC is not "92", the processor 81 determines NO in ACT 54 and proceeds to ACT 56. That is, after processing ACT 65 or if the output code OC is not "92", the processor 81 checks in ACT 56 whether the output code OC is "93".

[0113] If the output code OC is "93," the processor 81 determines YES in ACT56 and proceeds to ACT57. In ACT57, the processor 81 uses the function of the output unit 226 to output a third output command from the communication interface 87 to the self-checkout POS terminal 11. Specifically, the processor 81 obtains the register number from the time-series buffer 822 in which "93" is written as the output code OC. The processor 81 references the output table 823 and obtains the output data, i.e., sound data, in the same row as the output code OC "93." The third output command includes the register number and sound data.

[0114] The self-checkout POS terminal 11 outputs sound data to the speaker 85 of the self-checkout POS terminal 11 identified by the register number included in the third output command. After that, the processor 81 proceeds to ACT68.

[0115] If the output code OC is not "93", the processor 81 determines NO in ACT 56 and proceeds to ACT 58. That is, after processing ACT 57 or if the output code OC is not "93", the processor 81 checks in ACT 58 whether the output code OC is "94".

[0116] If the output code OC is "94," the processor 81 determines YES in ACT58 and proceeds to ACT59. In ACT59, the processor 81 uses the function of the output unit 226 to output a fourth output command from the communication interface 87 to the self-checkout POS terminal 11. Specifically, the processor 81 obtains the register number from the time-series buffer 822 in which "94" is written as the output code OC. The processor 81 references the output table 823 and obtains the output data, i.e., color data, in the same row as the output code OC "94." The fourth output command includes the register number and color data.

[0117] The self-checkout POS terminal 11 causes the light-emitting unit 65 of the self-checkout POS terminal 11 identified by the register number included in the fourth output command to emit light based on the color data. The light-emitting unit 65 may selectively emit a predetermined color or may blink. With this, the processor 81 ends its function as the output unit 226.

[0118] If the output code OC is not "94", the processor 81 determines NO in ACT58 and ends the function as the output unit 226.

[0119] As described above in detail, the processor 81 of the fraudulent activity estimation device 22 recognizes the picking up and bagging actions of a customer performing a product registration operation on the self-service POS terminal 11 based on the frame image captured by the camera 21, and calculates a recognition rate RP. The processor 81 writes the time TM, the action status AST, and the recognition rate RP corresponding to the frame image in the action file 821. The processor 81 calculates the reliability CD of the picking up and bagging actions based on the recognition rate RP written in the action file 821. The processor 81 writes the start time STM, the end time FTM, the action status AST, and the reliability CD in the time-series buffer 822. Furthermore, when the processor 81 recognizes a product registration operation based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14, it writes the current time TM as the operation status HST and the start time STM as operation information in the time-series buffer 822. The processor 81 outputs result information based on the operation information of the product registration operation and the reliability CD of the picking up and bagging actions.

[0120] For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the thresholds of the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12" written in the output table 823, respectively, the processor 81 obtains the output code OC and writes it together with the current time TM as the start time STM in the time series buffer 822. Then, the processor 81 outputs one or more output commands including output data corresponding to the output code OC in the output table 823 to the self-POS terminal 11 or the attendant terminal 14.

[0121] For example, when the processor 81 outputs a first output command including text data for the self-checkout POS terminal as output data to the self-checkout POS terminal 11, the text data for the self-checkout POS terminal notifying the customer of fraudulent activity is displayed on the touch panel 41 of the self-checkout POS terminal 11. This alerts the customer to the fact that fraudulent activity has occurred and enables the customer to be asked about the circumstances. As a result, fraudulent activity by customers against the self-checkout POS terminal 11 is deterred.

[0122] For example, when the processor 81 outputs a second output command including attendant terminal text data as output data to the attendant terminal 14, the attendant terminal text data notifying the attendant terminal of the fraudulent activity is displayed on the monitoring image SC of the attendant terminal 14. This allows the attendant to easily know that fraudulent activity has occurred. The attendant can quickly warn the customer and hear the details from the customer. As a result, fraudulent activity by customers at the self-service POS terminal 11 is deterred.

[0123] For example, when the processor 81 outputs a third output command including sound data as output data to the self-service POS terminal 11, sound data informing the customer of fraudulent behavior is output to the speaker 85 of the self-service POS terminal 11. This makes it possible to alert the customer to the fraudulent behavior. An attendant who hears the sound or voice can quickly alert the customer and hear the details from the customer. As a result, fraudulent behavior by customers against the self-service POS terminal 11 is deterred.

[0124] For example, when the processor 81 outputs a fourth output command including color data as output data to the self-service POS terminal 11, it causes the light-emitting unit 65 of the self-service POS terminal 11 to emit light based on the color data to alert the customer to fraudulent behavior. This allows an attendant who confirms that the light-emitting unit 65 is emitting light or flashing to quickly warn the customer and hear the details from them. As a result, fraudulent behavior by customers against the self-service POS terminal 11 is deterred.

[0125] In this way, when it is estimated that the reliability of the bagging of items that a customer has removed but not registered is high, output data is output according to the behavior status AST "11," i.e., the threshold value of the reliability CD of the removal behavior, and the behavior status AST "12," i.e., the threshold value of the reliability CD of the bagging behavior. These thresholds and output data can be set in advance by the store. Therefore, the fraud notification criteria, the recipients of fraud notification, the method of fraud notification, etc. can be set according to the store's policy.

[0126] For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action written in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "11" written in the output table 823, and / or the reliability CD of the bagging action written in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "12" written in the output table 823, the processor 81 records the action file 821 and the time series buffer 822 as a log.

[0127] In this way, if it is estimated that the reliability of a customer's bagging behavior of items that they removed but did not register is low, the behavior file 821 and the time-series buffer 822 are recorded as a log. Because no output data is output, a normal behavior of a customer is not mistakenly determined to be fraudulent, and a fraudulent behavior of a customer is not mistakenly determined to be normal. This prevents problems with customers and a decline in trust in the store. In addition, attendants can confirm whether a customer has committed fraud by analyzing the log. Furthermore, the behavior file 821 recorded as a log can be used in learning processes to improve the accuracy of skeleton estimation using frame images.

[0128] Therefore, according to this embodiment, it is possible to deal with the customer appropriately depending on the reliability of the customer's fraudulent behavior.

[0129] Although the embodiments of the misconduct estimation device, the control program therefor, and the misconduct estimation method have been described above, the embodiments are not limited to these.

[0130] In the above embodiment, an example has been given in which one camera 21 is arranged for one self-service POS terminal 11. A camera 21 does not necessarily have to be arranged for each self-service POS terminal 11. For example, if one camera 21 can capture an image of a customer operating two adjacent self-service POS terminals 11, the number of cameras 21 may be reduced. In that case, however, in ACT 2 of FIG. 8, the register number of the self-service POS terminal 11 closest to the position of the person shown in the captured image is acquired. Also, for example, multiple cameras 21 may be arranged for one self-service POS terminal 11. By doing so, blind spots of the self-service POS terminal 11 can be reduced and customer behavior can be recognized with high accuracy.

[0131] In the above embodiment, the processor 81 extracts the highest recognition rate RP from one or more recognition rates RP in which "11" is written as the action status AST, and sets this as the reliability CD of the take-out action. For example, when there are multiple recognition rates RP in which "11" is written as the action status AST, the processor 81 may calculate the average value of these. The processor 81 may set this average value as the reliability CD of the take-out action. Similarly, when there are multiple recognition rates RP in which "12" is written as the status ST of the bag-filling action, the processor 81 may calculate the average value of these. The processor 81 may set this average value as the reliability CD of the bag-filling action.

[0132] In the above embodiment, the removal action and the bagging action are exemplified as customer actions. For example, the processor 81 may recognize a registration action, a store-leaving action, etc. A registration action is an action of registering data of purchased items removed from a basket in the self-service POS terminal 11. For example, if the processor 81 detects a movement of the hand that performed the removal action, such as holding the purchased items over the reading window 42 in the center of the main body 40, the processor 81 recognizes that a registration action has occurred. Alternatively, if the processor 81 detects a movement of one hand that indicates an operation of the touch panel 41 of the main body 40, the processor 81 recognizes that a registration action has occurred. A store-leaving action is an action of a customer who has completed payment leaving the self-service POS terminal 11. For example, if the processor 81 detects a movement of the hand of a customer who has completed payment, such as removing a plastic bag or a personal bag from the holding arm 53, and then the customer can no longer be detected in the video captured by the camera 21, the processor 81 recognizes that a store-leaving action has occurred.

[0133] In the above embodiment, the output code OC "92" is text data for the attendant terminal. For example, the output code OC "92" may be text data and sound data for the attendant terminal. In this case, the text data for the attendant terminal is displayed on the monitoring image SC of the attendant terminal 14, and sound data is output from the speaker of the attendant terminal 14. The sound data may be, for example, a continuous sound or an intermittently repeated sound. The sound data may be, for example, a voice saying, "Fraud has been committed at register No. X." The sound data may be any sound or voice that notifies the attendant of a customer's fraudulent activity. In this case, the text data and sound data for the attendant terminal are examples of information that notifies the attendant of fraudulent activity. For example, in addition to output codes OC"91" to OC"94", output code OC"95" may be added to output table 823. The output data for output code OC"95" may be, for example, sound data for an attendant terminal. The sound data for an attendant terminal is an example of information that notifies of fraudulent activity. The sound data for an attendant terminal is an example of result information. In this case, the output data for output code OC"93" may be, for example, sound data for a self-service POS terminal. The sound data for a self-service POS terminal is an example of information that notifies of fraudulent activity. The sound data for a self-service POS terminal is an example of result information.

[0134] In the above embodiment, the processor 81 stores the behavior file 821 and the time series buffer 822 in a part of the non-volatile memory area of ​​the main memory 82 to be recorded as a log. For example, the processor 81 may transmit the behavior file 821 and the time series buffer 822 to the POS server 12 via the communication interface 87. The POS server 12 may store the behavior file 821 and the time series buffer 822 in the main memory or an auxiliary storage device. The behavior file 821 and the time series buffer 822 may be stored in the memory of an external device with which the fraudulent activity estimation device 22 can communicate, for example.

[0135] In the above embodiment, an example has been given of a case where, when "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the removal action written in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "11" written in the output table 823, and / or the reliability CD of the bagging action written in the time series buffer 822 does not satisfy the threshold of the reliability CD of the action status AST "12" written in the output table 823, the processor 81 records the action file 821 and the time series buffer 822 as a log. For example, even if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the thresholds of the reliability CD of the action status AST "11" and the reliability CD of the action status AST "12" written in the output table 823, respectively, the processor 81 may record the action file 821 and the time series buffer 822 as a log.

[0136] In the above embodiment, the output table 823 describes a threshold for the reliability CD of the behavior status AST "11" and a threshold for the reliability CD of the behavior status AST "12". For example, the output table 823 may be a data table that describes an output code OC, a first threshold for the reliability CD of the behavior status AST "11", a second threshold for the reliability CD of the behavior status AST "11", a first threshold for the reliability CD of the behavior status AST "12", a second threshold for the reliability CD of the behavior status AST "12", and output data in association with each other. The first and second thresholds may be set arbitrarily. The store may set the first and second thresholds for the reliability CD of the behavior status AST "11" and the first and second thresholds for the reliability CD of the behavior status AST "12" in advance. For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action and the reliability CD of the bagging action written in the time series buffer 822 satisfy the first threshold value of the reliability CD of the action status AST "11" and the first threshold value of the reliability CD of the action status AST "12" written in the output table 823, respectively, the processor 81 may obtain the output code OC and write it in the time series buffer 822 together with the current time TM as the start time STM. For example, if "22" is not written as the operation status HST in the time series buffer 822, and the reliability CD of the picking action written in the time series buffer 822 does not satisfy the first threshold value of the reliability CD of the action status AST "11" written in the output table 823 but satisfies the second threshold value, and / or the reliability CD of the bagging action written in the time series buffer 822 does not satisfy the first threshold value of the reliability CDD of the action status AST "12" written in the output table 823 but satisfies the second threshold value, the processor 81 may record the action file 821 and the time series buffer 822 as a log.

[0137] In the above embodiment, the attendant terminal 14 may incorporate the function of the display control device 13. In this case, the operation recognition unit 222 acquires data of the monitoring image SC from the attendant terminal 14 and recognizes the operation of the customer on the self-service POS terminal 11. Alternatively, the operation recognition unit 222 may acquire a data signal output from each self-service POS terminal 11 from the communication network 15 via a router, for example, and recognize the operation of the customer on the self-service POS terminal 11 based on the data signal.

[0138] In the above embodiment, the misconduct inference device 22 has functions as a behavior recognition unit 221, an operation recognition unit 222, a first acquisition unit 223, a second acquisition unit 224, a misconduct inference unit 225, and an output unit 226. The misconduct inference device 22 may be realized by a system in which functions are distributed among multiple devices.

[0139] In the above embodiment, the customer's skeleton is estimated from the image captured by the camera 21, and the customer's picking and bagging actions are recognized based on the temporal change in the correspondence between the hand positions and the regions of interest based on the skeleton estimation, such as the positions of the main body 40 of the self-service POS terminal 11, the bagging table 50, or the basket table 60. In this way, the behavior recognition method is not limited to the method based on skeleton estimation. 12, the processor 81 estimates the customer's left and right hands for each frame image captured by the camera 21, and infers a first bounding box 91 indicating the area of ​​the left hand and a second bounding box 92 indicating the area of ​​the right hand. The processor 81 also infers a third bounding box 93 indicating the area of ​​the basket 90 placed on the basket stand 60. The processor 81 selects the bounding box 91 or 92 indicating the area of ​​the hand holding the product from the first bounding box 91 or the second bounding box 92. For example, in FIG. 12, the second bounding box 92 is selected. The processor 81 tracks the movement of the first or second bounding box 91 or 92 selected from each frame image. The processor 81 also sets the third bounding box as a region of interest (ROI). The processor 81 then recognizes the customer's picking and bagging behaviors from the temporal change in the positional relationship between the first or second bounding box 91 or 92 and the ROI, which changes in accordance with the movement of the first or second bounding box 91 or 92. The region of interest is not limited to the third bounding box 93 indicating the region of the basket 90. For example, a fourth bounding box indicating the region of a shopping bag or a personal bag placed on the bagging table 50 may be inferred, and the customer's picking and bagging behavior may be recognized from the temporal change in the positional relationship between the movement of the first or second bounding box 91 or 92 and the region of interest.

[0140] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. The following is a summary of the scope of claims as originally filed in this application. [C1] a first acquisition means for acquiring operation information from a product registration operation performed by a purchaser on a payment terminal; behavior recognition means for recognizing the behavior of the purchaser who performs the product registration operation; A second acquisition means for acquiring the reliability of the behavior of the purchaser; an output means for outputting result information based on the operation information and the reliability; A fraudulent activity estimation device comprising: [C2] The fraudulent activity estimation device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means outputs information informing of fraudulent activity on the display unit of the payment terminal. [C3] The fraudulent activity inference device according to [C1], wherein when the reliability satisfies a predetermined condition, the output means outputs information informing a store clerk terminal of the fraudulent activity. [C4] The fraudulent activity estimation device according to [C1], wherein if the reliability satisfies a predetermined condition, the output means outputs a sound to the payment terminal to notify the user of fraudulent activity. [C5] The fraudulent activity estimation device according to [C1], wherein, when the reliability satisfies a predetermined condition, the output means causes a light-emitting unit of the payment terminal to emit light to indicate fraudulent activity. [C6] The fraudulent activity inference device according to [C1], wherein if the reliability does not satisfy a predetermined condition, the output means does not output the result information and stores the operation information and the reliability in a memory unit. [C7] The computer of the fraud estimation device A function to acquire operation information from the purchaser's product registration operation on the payment terminal, a function of recognizing the behavior of the purchaser who performs the product registration operation; A function for obtaining the reliability of the purchaser's behavior; and a function of outputting result information based on the operation information and the reliability; A control program to achieve this. [C8] The fraud estimation device Acquire operation information from the purchaser's product registration operation on the payment terminal, Recognizing the behavior of the purchaser who performs the product registration operation; Obtaining a reliability of the purchaser's behavior; outputting result information based on the operation information and the reliability. [C9] The behavior recognition means recognizes the behavior of the customer based on temporal changes in the correspondence between the position of the customer's hands based on their skeleton estimated from images taken by a camera and the position of the area of ​​interest related to the payment terminal.Fraudulent activity estimation device described in [C1]. [C10] The behavior recognition means recognizes the behavior of the purchaser based on temporal changes in the correspondence between the position of a bounding box indicating the area of ​​the purchaser's hand estimated from an image taken by a camera and the position of an area of ​​interest related to the payment terminal.Fraudulent activity estimation device described in [C1]. [Explanation of symbols]

[0141] 11...self-service POS terminal, 12...POS server, 13...display control device, 14...attendant terminal, 15...communication network, 21...camera, 22...fraudulent activity estimation device, 40...main body, 41...touch panel, 42...reading window, 43...card insertion slot, 44...receipt issuing slot, 45...coin insertion slot, 46...coin dispensing outlet, 47...banknote insertion slot, 48...banknote dispensing outlet, 50...bagging stand, 52...bag holder, 53...holding arm, 60...basket stand, 61...handheld scanner, 63...stand, 64...display pole, 65...light emitting unit, 81...processor, 82...main memory, 83...auxiliary storage device, 84...clock, 85...speaker, 86...camera interface, 87...communication interface, 88...system bus, 100...self-service POS system, 200...fraudulent activity estimation system, 821...behavior file, 822...time series buffer, 823...output table, 221...behavior recognition unit, 222...operation recognition unit, 223...first acquisition unit, 224...second acquisition unit, 225...fraudulent activity estimation unit, 226...output unit.

Claims

1. a first acquisition means for acquiring operation information from a product registration operation performed by a purchaser on a payment terminal; behavior recognition means for recognizing the behavior of the purchaser who performs the product registration operation; a second acquiring means for acquiring a reliability level representing the reliability of the recognition result of recognizing the behavior of the purchaser; a fraud inference means for inferring fraudulent behavior by the purchaser based on the operation information and the reliability; an output means for outputting result information for deterring fraudulent behavior by the purchaser based on the operation information and the reliability when fraudulent behavior by the purchaser is suspected; Equipped with The behavior recognition means is a fraudulent activity estimation device that recognizes the behavior of the purchaser based on temporal changes in the correspondence between the position of the purchaser's hand estimated from images taken by a camera and the position of the area of ​​interest related to the payment terminal.

2. 2. The fraudulent activity inference device according to claim 1, wherein, when the reliability satisfies a predetermined condition, the output means outputs information informing the user of fraud on a display unit of the payment terminal.

3. 2. The fraudulent activity inferring device according to claim 1, wherein, when the reliability satisfies a predetermined condition, the output means outputs information informing a store clerk terminal of the fraudulent activity.

4. 2. The fraudulent activity inference device according to claim 1, wherein, when the reliability satisfies a predetermined condition, the output means outputs a sound to the payment terminal to notify the user of fraud.

5. 2. The fraudulent activity inferring device according to claim 1, wherein, when the reliability satisfies a predetermined condition, the output means causes a light emitting unit of the payment terminal to emit light to indicate fraudulent activity.

6. 2. The misconduct inference device according to claim 1, wherein if the reliability does not satisfy a predetermined condition, the output means does not output the result information, and stores the operation information and the reliability in a storage unit.

7. The computer of the fraud estimation device A function to acquire operation information from the purchaser's product registration operation on the payment terminal, a function of recognizing the behavior of the purchaser who performs the product registration operation; A function of acquiring a reliability indicating the reliability of the recognition result of the purchaser's behavior; A function of inferring fraudulent behavior by the purchaser based on the operation information and the reliability; and a function of outputting result information for deterring fraudulent behavior by the purchaser based on the operation information and the reliability when fraudulent behavior by the purchaser is suspected; Realize this, A control program in which the function of recognizing the behavior of the purchaser performing the product registration operation is a function of recognizing the behavior of the purchaser based on temporal changes in the correspondence between the position of the purchaser's hand estimated from an image taken by a camera and the position of the area of ​​interest related to the payment terminal.

8. The fraud estimation device Acquire operation information from the purchaser's product registration operation on the payment terminal, Recognizing the behavior of the customer performing the product registration operation based on a temporal change in the correspondence between the position of the customer's hand estimated from the image captured by the camera and the position of the area of ​​interest related to the payment terminal; A reliability level representing the reliability of the recognition result of the purchaser's behavior is obtained; Inferring fraudulent behavior by the purchaser based on the operation information and the reliability; When fraudulent behavior by the purchaser is predicted, result information for deterring fraudulent behavior by the purchaser is output based on the operation information and the reliability.

9. The fraudulent activity estimation device described in claim 1, wherein the behavior recognition means estimates the customer's skeleton from images taken by a camera, and recognizes the customer's behavior based on temporal changes in the correspondence between the hand position based on the estimated skeleton and the position of the area of ​​interest related to the payment terminal.

10. The fraudulent activity estimation device described in claim 1, wherein the behavior recognition means estimates the customer's hand from an image captured by a camera and recognizes the customer's behavior based on temporal changes in the correspondence between the position of a bounding box indicating the estimated hand area and the position of an area of ​​interest related to the payment terminal.

Images