Fraud estimation device and program

Abstract

To provide a fraud estimation device and a program that send a notification to a clerk at a stage when a customer takes an action that is regarded as a precursor to fraudulent behavior.SOLUTION: A fraud estimation device includes a behavior recognition unit, a fraud estimation unit, and a control unit. The behavior recognition unit recognizes the behavior of a purchaser and outputs the recognition results. The fraud estimation unit estimates whether the purchaser has engaged in the behavior that is a precursor to the fraudulent behavior based on the recognition results. The control unit checks the estimation results of the fraud estimation unit, and controls output of a notification reporting the occurrence of the behavior that is a precursor to the fraudulent behavior when the purchaser has engaged in the behavior that is a precursor to the fraudulent behavior.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] An embodiment of the present invention relates to a fraud estimation device and a program. [Background technology]

[0002] In recent years, self-service POS (Point of Sales) terminals have been attracting attention in supermarkets and other retail stores from the perspective of reducing labor costs and taking measures to prevent the spread of infectious diseases. Self-service POS terminals are fully self-service payment terminals that allow customers to perform all operations from registering purchased items to making payments themselves. In order to monitor customers operating such self-service POS terminals, security cameras are often installed.

[0003] In a store system where such self-service POS terminals and surveillance cameras are installed, the behavior of a customer is recognized from the movement of the hand holding a purchased item, for example, based on the photographic data captured by the surveillance camera. The presence or absence of fraudulent activity is judged from the customer's behavior, and if fraudulent activity is determined, for example, a notification is given to a store clerk, a warning is issued on the self-service POS terminal, etc. However, by the time the clerk notices that fraudulent activity has been left unattended and arrives at the customer's location after determining that it is fraudulent, the customer who committed the fraudulent activity may have already left the register or left the store. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-89895 A Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the embodiments of the present invention is to provide a fraudulent activity estimation device and program that transmits a notification to a store clerk at the stage when a customer performs an action that is considered to be a sign of fraudulent activity. [Means for solving the problem]

[0006] In one embodiment, the fraudulent activity inference device includes a behavior recognition unit, a fraud inference unit, and a control unit. The behavior recognition unit recognizes the behavior of a purchaser and outputs the recognition result. The fraud inference unit infers from the recognition result whether the purchaser has engaged in behavior that is a precursor to fraudulent activity. The control unit checks the inference result of the fraud inference unit, and if the purchaser has engaged in behavior that is a precursor to fraudulent activity, controls to output a notification reporting the occurrence of behavior that is a precursor to fraudulent activity. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal is installed. [Diagram 2] FIG. 2 is a diagram for explaining the positional relationship between the self-service POS terminal and the camera. [Diagram 3] FIG. 3 is a schematic diagram showing an example of a monitoring image displayed on a display of an attendant terminal. [Figure 4] FIG. 4 is a block diagram showing a main circuit configuration of the misconduct inference device. [Diagram 5] FIG. 5 is a schematic diagram showing an example of the data structure of the behavior file. [Figure 6] FIG. 6 is a block diagram showing an example of the data structure of the POS terminal management file. [Figure 7] FIG. 7 is a flow diagram for explaining the functions of the behavior recognition unit and the second acquisition unit. [Figure 8] FIG. 8 is a flow chart for explaining the functions of the operation recognition unit and the first acquisition unit. [Figure 9] FIG. 9 is a flow chart for explaining the function of the fraud estimation unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] An embodiment will be described below with reference to the drawings. FIG. 1 is a system configuration diagram of a store in which a self-service POS terminal 11 is introduced. This system includes a self-service POS system 100 and a fraudulent activity estimation system 200. The self-service POS system 100 includes a plurality of self-service POS terminals 11, a POS server 12, a display control device 13, an attendant terminal 14, and a communication network 15. The plurality of self-service POS terminals 11, the POS server 12, and the display control device 13 are connected to the communication network 15. The attendant terminal 14 is connected to the display control device 13. The communication network 15 is typically a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

[0009] The self-service POS terminal 11 is a payment terminal for full self-service, which allows a customer to perform operations from registering purchased items to making a payment. A customer may also be called a purchaser, a consumer, a customer, or the like. The POS server 12 is a server computer for centrally controlling the operation of each self-service POS terminal 11. The display control device 13 is a controller that generates a monitoring image SC (see FIG. 3) for each self-service POS terminal 11 based on a data signal output from each self-service POS terminal 11 and displays the image on a display device of the attendant terminal 14. The attendant terminal 14 is a terminal for a store clerk called an attendant to monitor the status of each self-service POS terminal 11. The attendant terminal 14 is provided with a display such as a liquid crystal display or an organic EL display. The attendant terminal 14 divides the display screen into a plurality of sections, and displays a different monitoring image SC of the attendant terminal 14 for each section. The attendant terminal 14 is an example of a store clerk terminal. A conventionally known terminal can be directly applied to such a self-service POS system 100.

[0010] The fraudulent activity estimation system 200 includes a plurality of cameras 21 and a fraudulent activity estimation device 22. The plurality of cameras 21 correspond one-to-one to the plurality of self-service POS terminals 11, respectively. The cameras 21 are for photographing a customer who operates the corresponding self-service POS terminal 11.

[0011] The fraudulent activity estimation device 22 has functions as a behavior recognition unit 221, an operation recognition unit 222, a first acquisition unit 223, a second acquisition unit 224, a fraud estimation unit 225, and an output unit 226. The behavior recognition unit 221 is a function for recognizing the behavior of a customer who performs a product registration operation in the self-service POS terminal 11 based on the photographic data output from each camera 21. The behavior recognition unit 221 can also be called a behavior recognition means. The operation recognition unit 222 is a function for recognizing a product registration operation in the self-service POS terminal 11 by a customer based on the data of the monitoring image SC output from the display control device 13 to the attendant terminal 14. The operation recognition unit 222 can also be called an operation recognition means. The first acquisition unit 223 is a function for acquiring operation information from the recognized product registration operation. The first acquisition unit 223 can also be called a first acquisition means. The second acquisition unit 224 is a function for acquiring the recognition result by the behavior recognition unit 221. The second acquisition unit 224 can also be called a second acquisition means. The fraud estimation unit 225 is a function that estimates actions that are a precursor to fraud by a customer or fraudulent actions based on the operation information by the first acquisition unit 223 and the recognition results by the second acquisition unit 224. In this embodiment, an example of an action that is a precursor to fraud is picking up a bag (item) from the bagging area while registering a product. In this embodiment, an example of fraudulent actions is leaving the area imaged by the camera 21 while holding a product or bag without paying for it. The fraud estimation unit 225 can also be called fraud estimation means. The output unit 226 is a function that outputs estimated result information. The output unit 226 can also be called output means.

[0012] 2 is a diagram for explaining the positional relationship between the self-service POS terminal 11 and the camera 21. First, the external configuration of the self-service POS terminal 11 will be described.

[0013] The self-service POS terminal 11 comprises a main body 40 placed on the floor and a bagging stand 50 placed beside the main body 40. A touch panel 41 is attached to the top of the main body 40. The touch panel 41 is composed of a display and a touch sensor. The touch panel 41 is an example of a display unit. The display is a device for displaying various screens to an operator who operates the self-service POS terminal 11. The touch sensor is a device for detecting touch input to the screen by the operator. In the self-service POS terminal 11, the operator is usually a customer.

[0014] The main body 40 has a basket stand 60 in the center of the side opposite to the side where the bagging stand 50 is installed. The basket stand 60 is for customers coming from the sales floor to place baskets containing purchased items. Customers work by standing on the front side of the main body 40 in FIG. 2 so that they can see the screen of the touch panel 41. Therefore, when viewed from the customer, the basket stand 60 is on the right side of the main body 40, and the bagging stand 50 is on the left side. In this embodiment, the side where the customer stands is the front of the main body 40, the side where the bagging stand 50 is installed is the left side of the main body 40, and the side where the basket stand 60 is provided is the right side of the main body 40.

[0015] The main body 40 is provided with a scanner, a card reader, a receipt printer, a cash processing machine, etc. A scanner reading window 42, a card insertion port 43, a receipt issuing port 44, a coin insertion port 45, a coin dispensing port 46, a bill insertion port 47, and a bill dispensing port 48 are formed on the front side of the main body 40. A communication cable extends from the right side of the main body 40 to the outside, and a handy scanner 61 is connected to the tip of this communication cable. Although not shown, the main body 40 is also provided with a reader / writer for electronic money media.

[0016] A display pole 64 is attached to the upper surface of the main body 40. The display pole 64 has a light-emitting unit 65 at its tip. The light-emitting unit 65 selectively emits light, for example, in blue or red. The display pole 64 displays the status of the self-service POS terminal 11, such as waiting, in operation, calling, an error, a sign of fraud occurring, or a fraudulent act occurring, depending on the color of light emitted by the light-emitting unit 65. The display pole 64 may display the status of the self-service POS terminal 11 by causing the light-emitting unit 65 to flash.

[0017] The bagging table 50 has a structure in which a bag holder 52 is attached to its upper part. The bag holder 52 has a pair of holding arms 53, which hold a plastic bag provided in the store or a shopping bag brought by a customer, so-called a "my bag," or the like.

[0018] Next, a positional relationship between the self-service POS terminal 11 and the camera 21 will be described. As shown in Fig. 2, the camera 21 is installed in a position where it can photograph a customer standing in front of the self-service POS terminal 11 and facing the main body 40, the bagging table 50, the basket table 60, and other components from above.

[0019] A customer standing in front of the self-service POS terminal 11 first places a basket containing purchased items on the basket stand 60 on the right side, and places a shopping bag or a personal bag on the holding arm 53 on the left side. Next, the customer follows the guidance displayed on the touch panel 41 and operates the touch panel 41 to declare the start of use of the self-service POS terminal 11.

[0020] The customer then picks up the items to be purchased one by one from the baskets placed on the basket stand 60. If the purchased items have barcodes, the customer registers the items by holding the barcode over the reading window 42 and having the barcode read by the scanner. If the purchased items do not have barcodes, the customer registers the items by operating the touch panel 41 to select the purchased item from a list of items without barcodes. After registering the items, the customer places them in a shopping bag, a reusable bag, or the like.

[0021] After registering all purchased items, the customer operates touch panel 41 to select a payment method. For example, if cash payment is selected, the customer inserts bills or coins into bill slot 47 or coin slot 45, and removes the change from bill payout outlet 48 or coin payout outlet 46. For example, if electronic money payment is selected, the customer holds the electronic money medium over the reader / writer. For example, if credit card payment is selected, the customer inserts a credit card into card slot 43. Having completed payment in this manner, the customer receives the receipt issued from receipt issuing port 44, and leaves the store with the shopping bag or reusable bag removed from holding arm 53.

[0022] That is, the camera 21 is installed in a position in front of the self-service POS terminal 11 so as to be able to capture the hand movements of a customer who behaves as described above.

[0023] Fig. 3 is a schematic diagram showing an example of a monitoring image SC displayed on the display of the attendant terminal 14. As described above, the monitor images SC for the multiple self POS terminals 11 are displayed separately on the display of the attendant terminal 14. Fig. 3 shows an example of a monitoring image SC for one of the self POS terminals 11. The configuration of the monitoring image SC for the other self POS terminals 11 is similar, so a description thereof will be omitted here.

[0024] As shown in FIG. 3, the surveillance image SC includes a register number column 71, a terminal status column 72, an error information column 73, a declaration information column 74, a details column 75, and a total column .

[0025] The register number column 71 is a column for displaying the register number. The register number is a series of numbers that are assigned to each self-service POS terminal 11 without duplication in order to individually identify each self-service POS terminal 11. The register number is identification information for identifying each self-service POS terminal 11.

[0026] The terminal status field 72 is a field for displaying the operation status of the self-service POS terminal 11. For example, any one of "standby", "start of use", "registering", "start of payment", and "payment in progress" is displayed in the terminal status field 72 as the operation status.

[0027] "Waiting" is the state from when the previous customer has finished payment until the next customer announces that they are ready to use the service. An initial image is displayed on the touch panel 41 of the self-service POS terminal 11 in the "Waiting" state. The initial image is, for example, an image including a touch button that allows the customer to select whether to use a plastic bag provided in the store or to use their own bag.

[0028] "Start of use" is the state where a customer standing in front of the self-service POS terminal 11 declares the start of use for payment. The customer selects on the initial image whether to use a shopping bag or a reusable bag. This selection operation declares the start of use. In response to this selection operation, the operating state of the self-service POS terminal 11 becomes "start of use".

[0029] "Registering" is a state in which the self-service POS terminal 11 is accepting the customer's registration of the purchased item. When the first purchased item is registered, the operation status of the self-service POS terminal 11 becomes "Registering". After that, the operation status of the self-service POS terminal 11 remains "Registering" until the transition to payment is declared.

[0030] "Payment started" is the state where a customer who has finished registering the items to be purchased declares the transition to payment. The "Checkout" soft key is displayed on the touch panel 41 of the self-service POS terminal 11 in the "Registering" state. The customer who has finished registering the items to be purchased touches the "Checkout" soft key. This operation declares the transition to payment. In response to this operation, the operating state of the self-service POS terminal 11 becomes "Payment started."

[0031] "Payment in progress" is a state in which payment processing such as cash payment, electronic money payment, credit card payment, etc. is being performed. For example, when a bill or coin is inserted into the bill insertion slot 47 or the coin insertion slot 45, the operation status of the self-service POS terminal 11 becomes "payment in progress". Then, when the payment processing is completed, the operation status of the self-service POS terminal 11 returns to "standby".

[0032] The error information field 73 is a field for displaying error information that has occurred in the self-service POS terminal 11. Error information includes communication errors, receipt-out errors, etc. The declaration information field 74 is a field for displaying the contents of the customer's declaration operation. For example, if a customer selects "my bag," "no bag needed" is displayed, indicating that a plastic bag is not required.

[0033] The details column 75 is a column for displaying detailed information of purchased items registered at the self-service POS terminal 11. The detailed information is, for example, the product name, number of items, amount, etc. of the purchased items. The total column 76 is a column for displaying total information of purchased items registered at the self-service POS terminal 11. The total information is, for example, the total number of items, the total amount, the amount inserted, change, etc.

[0034] It should be noted that the configuration of the surveillance image SC is not limited to this. Columns for displaying other items may be arranged. Furthermore, the items of text data displayed in FIG. 3 are not limited to this. Text data of other items may be displayed. Furthermore, actions that are a sign of fraudulent activity or notifications that fraudulent activity has occurred may be displayed as a separate image that is displayed as part of or in front of the surveillance image SC.

[0035] 4 is a block diagram showing a main circuit configuration of the misconduct estimation device 22. The misconduct estimation device 22 includes a processor 81, a main memory 82, an auxiliary storage device 83, a clock 84, a speaker 85, a camera interface 86, a communication interface 87, and a system bus 88. The system bus 88 includes an address bus, a data bus, and the like. The misconduct estimation device 22 configures a computer by connecting the processor 81, the main memory 82, the auxiliary storage device 83, the clock 84, the speaker 85, the camera interface 86, and the communication interface 87 via the system bus 88.

[0036] The processor 81 corresponds to the central part of the computer. The processor 81 controls each part to realize various functions of the misconduct estimation device 22 according to an operating system or an application program. The processor 81 is, for example, a CPU (Central Processing Unit).

[0037] The main memory 82 corresponds to the main storage portion of the computer. The main memory 82 includes a nonvolatile memory area and a volatile memory area. In the nonvolatile memory area, the main memory 82 stores an operating system or an application program. In the volatile memory area, the main memory 82 stores data required for the processor 81 to execute processes for controlling each part. This type of data may also be stored in the nonvolatile memory area. The main memory 82 uses the volatile memory area as a work area where data is appropriately rewritten by the processor 81. The nonvolatile memory area is, for example, a ROM (Read Only Memory). The volatile memory area is, for example, a RAM (Random Access Memory).

[0038] The auxiliary storage device 83 corresponds to the auxiliary storage portion of the computer. As the auxiliary storage device 83, for example, a well-known storage device such as an SSD (Solid State Drive), an HDD (Hard Disc Drive), or an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory) is used alone or in combination. The auxiliary storage device 83 stores data used by the processor 81 when performing various processes, data generated by the processes in the processor 81, etc. The auxiliary storage device 83 may also store an application program.

[0039] The application programs stored in the main memory 82 or the auxiliary storage device 83 include a control program, which will be described later. The method of installing the control program in the main memory 82 or the auxiliary storage device 83 is not particularly limited. The control program can be installed in the main memory 82 or the auxiliary storage device 83 by recording it on a removable recording medium, or by distributing the control program by communication via a network. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it can store the program and can be read by the device. In addition, the main memory 82 or the auxiliary storage device 83 stores a notification to be sent to the attendant terminal 14 when a customer performs an action that is a sign of fraudulent behavior or an action that is fraudulent behavior. The notification is composed of a text message, a voice file, the light emission color or light emission pattern of the display pole 64, or the like.

[0040] The clock 84 functions as a time information source for the fraudulent activity estimation device 22. The processor 81 obtains the current date and time based on the time information kept by the clock 84.

[0041] The speaker 85 is an output device for outputting sound data. The sound data includes sounds, voices, and the like.

[0042] The camera interface 86 is an interface for communicating with each camera 21. The photographed data output from each camera 21 is taken into the fraudulent activity estimation device 22 via the camera interface 86. The photographed data includes a photographed video, a photographed image, etc., of a customer operating the self-service POS terminal 11 corresponding to the camera 21.

[0043] The communication interface 87 is an interface for performing data communication in accordance with a communication protocol among the self-service POS terminal 11, the POS server 12, the display control device 13, etc. For example, image data output from the display control device 13 is input into the fraudulent activity estimation device 22 via the communication interface 87. The image data is data of a surveillance image SC generated for each self-service POS terminal 11.

[0044] The misconduct estimation device 22 configured as above uses a part of the volatile memory area in the main memory 82 as an area for the behavior file 821 and the POS terminal management file 822. The misconduct estimation device 22 then forms the behavior file 821 and the POS terminal management file 822 in this area.

[0045] Fig. 5 is a schematic diagram showing an example of the data structure of the action file 821. As shown in Fig. 5, the action file 821 is a data file in which a time TM, an action status AST, a recognition rate RP, and a frame image are recorded in association with each other for each register number that identifies the self-service POS terminal 11. The time TM is the time at which the action status AST is acquired. The action status AST represents a state in which the customer's action is recognized by the function of the action recognition unit 221 of the processor 81, which will be described later. In this embodiment, the customer's actions are a removal action, a bagging action, a fraudulent action precursor action, and a fraudulent action.

[0046] The take-out action is an action of taking out a purchased item from a basket placed on the basket stand 60. For example, when the skeleton of one or both hands moves to the right side of the main body 40 and a movement of lifting up a purchased item is detected, the processor 81 recognizes that a take-out action has occurred. When the processor 81 recognizes a take-out action, it enters "11" in the action status AST.

[0047] The bagging action is an action of placing registered purchased items into a shopping bag or a reusable bag on the bagging table 50. For example, when the skeleton of the hand that performed the registered action moves to the left side of the main body and a movement of placing the purchased items into a shopping bag or a reusable bag is detected, the processor 81 recognizes that a bagging action has occurred. When the processor 81 recognizes a bagging action, it enters "12" into the action status AST.

[0048] A fraudulent behavior precursor behavior is a behavior of lifting a shopping bag, a reusable bag, or the like from the bagging table 50 during product registration, which is a precursor to fraudulent behavior. For example, when the processor 81 detects a movement of one or both hands grabbing and lifting a shopping bag, a reusable bag, or a product during product registration or before payment is completed, the processor 81 recognizes that a fraudulent behavior precursor behavior has occurred. When the processor 81 recognizes a fraudulent behavior precursor behavior, it enters "13" into the behavior status AST.

[0049] Fraudulent behavior is a behavior in which a customer takes a shopping bag, a personal bag, or a product out of the imaging range of camera 21 and leaves before completing payment. For example, if the skeleton of one or both hands holding a shopping bag, a personal bag, or a product cannot be detected from the imaging range of camera 21 before completing payment, processor 81 recognizes that fraud has occurred. When processor 81 recognizes fraud, it writes "14" in the behavior status AST.

[0050] The recognition rate RP is a numerical value calculated based on the recognition rate at which the processor 81 recognizes the customer's behavior included in the captured image as a removal behavior, the recognition rate at which the processor 81 recognizes the customer's behavior as a bagging behavior, and the recognition rate at which the processor 81 recognizes the customer's behavior as a precursor to fraudulent activity. The recognition rate RP is, for example, a percentage. The frame images are images captured by the camera 21. The frame images are recorded in the behavior file 821 in the order in which they were captured. Note that the items displayed in FIG. 5 are not limited to these. Other items may be displayed. The contents of the text data displayed in FIG. 5 are an example.

[0051] 6 is a schematic diagram showing an example of the POS terminal management file 822. As shown in FIG. 6, the POS terminal management file 822 has an area for describing a start time STM, an end time FTM, and a status ST in association with each other for each register number that identifies the self-service POS terminal 11. The start time STM is the time TM at which the behavior status AST is described earliest in the behavior file 821. The start time STM is the time at which the operation status HST is acquired. The end time FTM is the time TM at which the behavior status AST is described latest in the behavior file 821. The status ST includes the behavior status AST and the operation status HST. The operation status HST indicates a state in which a product registration operation by a customer to the self-service POS terminal 11 is recognized by the function of the operation recognition unit 222 of the processor 81 described later.

[0052] The functions of the misconduct inference device 22 as the behavior recognition unit 221, the operation recognition unit 222, the first acquisition unit 223, the second acquisition unit 224, the fraud inference unit 225, and the output unit 226 are functions provided for each self POS terminal 11. The functions of the behavior recognition unit 221, the operation recognition unit 222, the first acquisition unit 223, the second acquisition unit 224, the fraud inference unit 225, and the output unit 226 for other self POS terminals 11 are similar. The misconduct inference device 22 realizes the functions of the behavior recognition unit 221, the operation recognition unit 222, the first acquisition unit 223, the second acquisition unit 224, the fraud inference unit 225, and the output unit 226 by a processor 81 and a control program that controls the processor 81.

[0053] 7 to 9 are flow charts showing the main control procedure of the processor 81 in the fraudulent activity estimation device 22. Below, using these flow charts, the main operations of the system of the store in which the self-service POS terminal 11 is installed will be explained. Note that the operations explained below are just an example. As long as similar results are obtained, the procedure is not particularly limited.

[0054] 7 is a flow chart for explaining the functions of the behavior recognition unit 221 and the second acquisition unit 224. The processor 81 waits to recognize a customer (ACT1). The camera 21 is installed in a position where the camera 21 can capture an image of a customer standing in front of the self-service POS terminal 11 from above. When the processor 81 detects from the image captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, the processor 81 determines that the customer has been recognized.

[0055] When the processor 81 recognizes the customer (YES in ACT1), the process proceeds to ACT2. The processor 81 acquires the register number of the self-service POS terminal 11 (ACT2). Each camera 21 corresponds one-to-one with each self-service POS terminal 11. The processor 81 identifies the self-service POS terminal 11 from the identification information of the camera 21 capturing an image of a customer standing in front of the self-service POS terminal 11, and acquires the register number of the self-service POS terminal 11.

[0056] The processor 81 starts acquiring frame images captured by the camera 21 (ACT3). The processor 81 stores the frame image in the behavior file 821 every time the processor 81 acquires a frame image.

[0057] The processor 81 estimates the skeleton of a person, that is, a customer, from the frame image captured by the camera 21 (ACT4). Skeleton estimation can be realized even with an inexpensive camera 21 by using AI technology such as deep learning. The processor 81 recognizes the customer's take-out action or bagging action from the hand movement obtained by the skeleton estimation and the temporal change in the positional relationship between the hand and a region of interest (ROI) that changes with the movement, for example, the main body 40 of the self-POS terminal 11, the bagging table 50, or the basket table 60, and calculates the recognition rate RP of each. Specifically, the processor 81 calculates the recognition rate of the take-out action and the recognition rate of the bagging action from the hand movement. The processor 81 sets the highest recognition rate among these recognition rates as the recognition rate RP. For example, if the recognition rate of the take-out action is 98 percent and the recognition rate of the bagging action is 2 percent, the recognition rate RP is 98 percent. The processor 81 recognizes the take-out action for which the recognition rate RP of 98 percent is calculated as the customer's action. The process of calculating the recognition rate RP based on hand movements is a known process, and therefore a detailed description thereof will be omitted.

[0058] The processor 81 checks whether or not the take-out behavior has been recognized based on the processing of ACT4 (ACT5). If the take-out behavior has been recognized by the function of the behavior recognition unit 221 (YES in ACT5), the processor 81 advances the processing to ACT6.

[0059] The processor 81 sets the action status AST to "11" (ACT6). After that, the processor 81 advances the process to ACT9.

[0060] Processor 81 acquires the current time TM measured by clock 84 (ACT9). Processor 81 writes the time TM, the action status AST of "11", and the recognition rate RP calculated in the process of ACT4 in association with the frame image saved in the process of ACT3 in the action file 821 in which the register number acquired in the process of ACT2 is set (ACT10). After that, processor 81 returns the process to ACT3.

[0061] The processor 81 inputs the start time STM or end time FTM and the status ST to the POS terminal management file 822 (ACT11). The processor 81 writes the current time TM acquired in the processing of ACT9 in the start time STM or end time FTM, and writes "11" in the status ST.

[0062] If the take-out action is not recognized (NO in ACT5), the processor 81 advances the process to ACT7.

[0063] The processor 81 checks whether or not a bagging action has been recognized based on the processing of ACT4 (ACT7). If a bagging action has been recognized by the function of the behavior recognition unit 221 (YES in ACT7), the processor 81 advances the processing to ACT8.

[0064] The processor 81 sets the action status AST to "12" (ACT8).

[0065] The processor 81 obtains the current time TM measured by the clock 84 (ACT9). The processor 81 writes the time TM, the action status AST of "12", and the recognition rate RP calculated in the process of ACT4 in association with the frame image saved in the process of ACT3 in the action file 821 (ACT10). After that, the processor 81 returns the process to ACT3.

[0066] The processor 81 writes the start time STM or end time FTM and the status ST in the POS terminal management file 822 (ACT11). The processor 81 writes the current time TM acquired in the processing of ACT9 in the start time STM or end time FTM, and writes "12" in the status ST.

[0067] If the processor 81 does not recognize the bagging behavior (NO in ACT7), it checks whether the behavior is a precursor to fraudulent activity (ACT12). The processor 81 checks whether it has recognized a precursor to fraudulent activity based on the processing in ACT4. If the processor 81 has not recognized a precursor to fraudulent activity (NO in ACT12), it ends the processing.

[0068] If the behavior is recognized as a precursor to fraudulent activity (YES in ACT12), the processor 81 sets the behavior status AST to "13" (ACT13).

[0069] The processor 81 obtains the current time TM measured by the clock 84 (ACT14). The processor 81 writes in the behavior file 821 the time TM, the behavior status AST of "13", and the recognition rate RP calculated in the processing of ACT4 in association with each other, in correspondence with the frame image saved in the processing of ACT3 (ACT15).

[0070] The processor 81 writes the start time STM and the status ST in the POS terminal management file 822 (ACT16). The processor 81 writes the current time TM acquired in the process of ACT9 in the start time STM, and writes "13" in the status ST.

[0071] After completing ACT13, the processor 81 ends this process and transitions to a process related to the function as the fraud estimation unit 225, which will be described later.

[0072] Thereafter, when it is detected again from the image captured by the camera 21 that a person is standing in front of the self-service POS terminal 11, the processor 81 executes the processes of ACT2 to ACT10 in the same manner as described above.

[0073] 8 is a flow chart for explaining the functions of the operation recognition unit 222 and the first acquisition unit 223. The processor 81 waits for the start of use to be declared for the self-service POS terminal 11 (ACT21). When the start of use is declared, "Start of use" is displayed in the terminal status field 72 of the monitoring image SC corresponding to the self-service POS terminal 11. The processor 81 checks whether the characters "Start of use" can be recognized from the terminal status field 72 of the monitoring image SC acquired via the display control device 13. When the characters "Start of use" can be recognized, the processor 81 recognizes that the start of use has been declared by the function of the operation recognition unit 222.

[0074] When it is recognized that the start of use has been declared (YES in ACT21), the processor 81 proceeds to the process in ACT22. The processor 81 acquires the register number of the self-service POS terminal 11 (ACT22). The register number is displayed in the register number column 71 of the monitoring image SC. The processor 81 recognizes the characters of the register number from the register number column 71 of the monitoring image SC acquired via the display control device 13, and acquires the characters as the register number.

[0075] The processor 81 sets the operation status HST to "21" (ACT23). The operation status HST "21" indicates a start operation. The processor 81 acquires the current time TM measured by the clock 84 (ACT24). The processor 81 associates the time TM as the start time STM with "21" as the operation status HST and writes it in the POS terminal management file 822 in which the register number acquired in the processing of ACT22 is set (ACT25). The processor 81 also writes a NULL value in the end time FTM.

[0076] Therefore, when a customer standing in front of the self-service POS terminal 11 performs a declaration operation to start use, first, "21" is written as the operation status HST together with the time TM in the POS terminal management file 822 corresponding to that self-service POS terminal 11.

[0077] The processor 81 starts recognizing an operation on the self-service POS terminal 11 (ACT26). Specifically, the processor 81 recognizes, for example, a product registration operation, a payment start operation, etc., from the transition of information obtained by character recognition of the monitoring image SC acquired via the display control device 13.

[0078] For example, processor 81 recognizes that a product registration operation has been performed when detailed information such as the product name, number, and price of a purchased product is added to detail column 75. For example, processor 81 recognizes that a payment start operation has been performed when the display in terminal status column 72 switches to "Payment Start."

[0079] The processor 81 waits to recognize a product registration operation or a payment start operation as ACT27 or ACT28.

[0080] In the standby state of ACT27 or ACT28, when a product registration operation is recognized by the function of the operation recognition unit 222 (YES in ACT27), the processor 81 proceeds to ACT29.

[0081] The processor 81 sets the operation status HST to "22" by the function of the first acquisition unit 223 (ACT29). The operation status HST "22" indicates a product registration operation. The processor 81 acquires the current time TM measured by the clock 84 by the function of the first acquisition unit 223 (ACT30). The processor 81 associates the time TM as the start time STM with "22" as the operation status HST and writes it in the POS terminal management file 822 (ACT31). The processor 81 writes a NULL value in the end time FTM. The operation status HST is an example of operation information. The current time TM is an example of operation information. The processor 81 then returns to the standby state of ACT27 or ACT28.

[0082] In the standby state of ACT27 or ACT28, when the operation recognition unit 222 recognizes a payment start operation (YES in ACT28), the processor 81 advances the process to ACT32. The processor 81 ends the operation recognition for the self POS terminal 11 (ACT32).

[0083] The processor 81 sets the operation status HST to "23" (ACT33). The operation status HST "23" indicates a payment start operation. The processor 81 acquires the current time TM measured by the clock 84 (ACT34). The processor 81 associates the time TM as the start time STM with "23" as the operation status HST and writes it in the POS terminal management file 822 (ACT35). The processor 81 writes a NULL value in the end time FTM. With this, the processor 81 terminates its functions as the operation recognition unit 222 and the first acquisition unit 223.

[0084] Thereafter, when the processor 81 detects again from the data of the monitoring image SC an operation to start use of the self-service POS terminal 11, the processor 81 executes the processes of ACT22 to ACT35 in the same manner as described above.

[0085] Fig. 9 is a flow chart for explaining the function of the fraud estimation unit 225. In this embodiment, the processor 81 executes the process of the procedure shown in Fig. 10 for each purchased item.

[0086] When the processor 81 does not recognize the bagging behavior (NO in ACT7), it checks whether or not "22" is described as the operation status HST in the POS terminal management file 822 (ACT41). Specifically, the processor 81 checks the POS terminal management file 822, and checks whether or not "22" is described as the operation status HST in association with the time TM as the start time STM before the start time STM at which "13" is described as the behavior status AST. When "22" is described as the operation status HST (YES in ACT41), the processor 81 deletes the behavior status AST "13" from the POS terminal management file 822 (ACT42). After deleting the behavior status AST "13", the processor 81 ends its function as the fraud estimation unit 225.

[0087] If "22" is not written as the operation status HST (NO in ACT41), the processor 81 uses the function of the output unit 226 to output a notice regarding a sign of fraudulent activity from the communication interface 87 to the attendant terminal 14 via the display control device 13 (ACT43). The processor 81 accesses the main memory 82 or the auxiliary storage device 83, and outputs a notice to be sent to a store clerk when it detects an action that is a sign of fraudulent activity, the notice being composed of a text message, an audio file, the light emission color or light emission pattern of the display pole 64, or the like.

[0088] The processor 81 checks whether the customer has performed a payment start operation (ACT44). For example, the processor 81 recognizes the payment start operation, etc., from the transition of information obtained by character recognition of the monitoring image SC acquired via the display control device 13. If it is confirmed as a result of the check that the customer has performed a payment start operation (YES in ACT44), the processor 81 deletes the action status AST "13" from the POS terminal management file 822 (ACT42) and ends the process.

[0089] If it is not confirmed that the customer has performed the payment initiation operation (ACT44: NO), the processor 81 checks whether the customer has committed fraud (ACT45). The processor 81 performs the same processing as in ACT3 and ACT4, and checks whether the customer's fraudulent activity has been recognized based on the results. If the processor 81 is unable to recognize fraudulent activity using the function of the behavior recognition unit 221 (ACT45: NO), the processor 81 deletes the behavior status AST "13" from the POS terminal management file 822 (ACT42) and ends the process.

[0090] If the confirmation result indicates that fraudulent activity has occurred (YES in ACT45), the processor 81 acquires the current time TM measured by the clock 84 (ACT46). The processor 81 writes the time TM, the action status AST of "14", and the recognition rate RP calculated in the processing of ACT45 in association with the frame image saved in the processing of ACT45 in the action file 821 (ACT47). After that, the processor 81 writes the action status AST "14" in the POS terminal management file 822 (ACT48). The processor 81 writes the start time STM and "14" in the status ST based on the time TM acquired in ACT46 (ACT49). The processor 81 writes a NULL value for the end time FTM.

[0091] The processor 81 uses the function of the output unit 226 to output a notification that fraudulent activity has occurred from the communication interface 87 to the attendant terminal 14 via the display control device 13 (ACT50). The processor 81 accesses the main memory 82 or the auxiliary storage device 83, and outputs a notification to be sent to a store clerk when fraudulent activity is detected, the notification being composed of a text message, an audio file, the light emission color or light emission pattern of the display pole 64, or the like, and then ends.

[0092] According to the above embodiment, by notifying a store clerk at the stage when a customer exhibits a warning sign before committing fraud, it is possible to provide the store with time to take action before the fraud is committed.

[0093] Although the embodiment of the misconduct estimation device has been described above, the embodiment is not limited to this.

[0094] In the above embodiment, a case where one camera 21 is arranged for one self-service POS terminal 11 is exemplified. The camera 21 does not necessarily have to be arranged for each self-service POS terminal 11. For example, if a customer operating two adjacent self-service POS terminals 11 can be photographed by one camera 21, the number of cameras 21 may be reduced. In that case, however, in ACT2 of FIG. 8, the register number of the self-service POS terminal 11 closest to the position of the person shown in the photographed image is obtained. Also, for example, multiple cameras 21 may be arranged for one self-service POS terminal 11. By doing so, the blind spot of the self-service POS terminal 11 can be reduced, and the behavior of the customer can be recognized with high accuracy.

[0095] In the above embodiment, the processor 81 judges whether the customer has performed a precursor to fraudulent behavior from the customer's behavior, but the condition may include the duration of the behavior. For example, a process may be provided to judge whether a behavior other than removal or bagging has been performed for a preset period of time between NO in ACT7 and the transition to ACT12. Specifically, if a behavior other than removal or bagging is set to be regarded as a precursor to fraudulent behavior if performed for 10 seconds, the processor 81 waits for 10 seconds after the occurrence of the behavior other than removal or bagging, and checks from the frame image 10 seconds later whether the behavior other than removal or bagging has been continued. Even in such a case, the store clerk can be notified of the precursor to fraudulent behavior by the customer, and the accuracy of detecting the precursor to fraudulent behavior can be improved by not regarding the behavior of the customer simply moving a bag as a precursor to fraudulent behavior.

[0096] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0097] 11...self-service POS terminal, 12...POS server, 13...display control device, 14...attendant terminal, 15...communication network, 21...camera, 22...fraudulent activity estimation device, 40...main body, 41...touch panel, 42...reading window, 43...card insertion port, 44...receipt issuing port, 45...coin insertion port, 46...coin dispensing outlet, 47...banknote insertion port, 48...banknote dispensing outlet, 50...bag filling table, 52...bag holder, 53...holding arm, 60...basket stand, 61...handheld scanner, 63...placement stand, 64...display Pole, 65...light emitting unit, 81...processor, 82...main memory, 83...auxiliary storage device, 84...clock, 85...speaker, 86...camera interface, 87...communication interface, 88...system bus, 100...self-service POS system, 200...fraudulent activity estimation system, 821...behavior file, 822...POS terminal management file, 221...behavior recognition unit, 222...operation recognition unit, 223...first acquisition unit, 224...second acquisition unit, 225...fraudulent activity estimation unit, 226...output unit.

Claims

1. a behavior recognition unit that recognizes a behavior of a purchaser and outputs a recognition result; a fraud estimation unit that estimates, from the recognition result, whether the purchaser has engaged in a behavior that is a precursor to a fraudulent act that has been set in advance; a control unit that, when the fraud estimation unit determines that the purchaser has engaged in a behavior that is a precursor to fraud, outputs a notification that the behavior that is a precursor to fraud has occurred; and A fraudulent activity estimation device comprising:

2. the fraud estimation unit estimates whether a behavior that constitutes a predetermined fraudulent act has been performed after estimating the behavior that is a precursor to the fraudulent act, The fraudulent activity estimation device according to claim 1 , wherein the control unit performs control to output a notification reporting that the fraudulent activity has occurred when the fraud estimation unit determines that the purchaser has engaged in behavior that constitutes the fraudulent activity.

3. Further comprising a clock unit for acquiring the current time, The fraudulent activity inference device according to claim 1 , wherein the control unit associates the behavior that is a precursor of the fraudulent activity with the current time acquired by the clock unit.

4. The computer that detects fraudulent activity The ability to capture buyer behavior, A function of recognizing the behavior of the purchaser and outputting the recognition result; A function of inferring whether the purchaser has engaged in any behavior that is a precursor to fraudulent activity based on the recognition result and outputting the inference result; a function of notifying the purchaser of the occurrence of a behavior that is a precursor to fraudulent activity when the result of the estimation indicates that the purchaser has engaged in a behavior that is a precursor to fraudulent activity; A program to achieve this.

Images